Online Data Processing at ID31#

Workflows can be triggered automatically for every scan that has a diffraction camera that needs XRPD data reduction.

Usage#

The automatic XRPD data reduction workflows can be enabled through the xrpd_processor object

SIXC [1]: xrpd_processor.enable(p3)
SIXC [2]: xrpd_processor.options["pyfai_config"] = "/data/visitor/a311182/bm31/20221002/Si_125.poni"
SIXC [3]: xrpd_processor.options["retry_timeout"] = 3600
SIXC [4]: xrpd_processor.options["monitor_name"] = "mondio"
SIXC [5]: xrpd_processor.options["reference"] = 1

SIXC [6]: xrpd_processor
Out [6]: XRPD processing:
        enabled               True
        integration_options   {'method': 'csr_ocl_gpu'}
        detector_name         p3
        counter_names         ['mondio', 'detdio']
        pyfai_config          /data/visitor/a311182/bm31/20221002/Si_125.poni
        retry_timeout         3600

The configuration can be a poni file or json file generate manually with the pyfai GUI tools.

Currently all results will be saved in the proposal folder:

/data/visitor/<proposal>/bm31/<session>/processed

Disable online data processing

SIXC [6]: xrpd_processor.disable()
SIXC [7]: xrpd_processor
Out [7]: XRPD processing:
        enabled               False
        integration_options   {'method': 'csr_ocl_gpu'}
        detector_name         p3
        counter_names         ['mondio', 'detdio']
        pyfai_config          /data/visitor/a311182/bm31/20221002/Si_125.poni
        retry_timeout         3600
        monitor_name          'mondio'
        reference             1

Installation#

Installation on lid31dat1 (data processing)#

One conda environment called ewoksworker

Dependencies installed:

pip install ewoksjob[blissworker]
pip install ewoksserver[frontend]
pip install ewoksxrpd
pip install blissoda

Processes registered with the supervisor

  • ewoksmonitor: monitor data processing http://lid31dat1:5555/

  • ewoksserver: create workflows http://lid31dat1:5000/

  • ewoksworker: celery worker that does the actual calculation

  • sixcprocessor: listens to the SIXC session and sends workflow to the worker

Installation on bibhelm (acquisition control)#

Add ewoks configuration to beacon.

Dependencies installed in the bliss environment

pip install blissoda[id31]

Add this to every session setup script that needs online data processing

from blissoda.id31.xrpd_processor import XrpdProcessor as _XrpdProcessor
xrpd_processor = _XrpdProcessor()