Online Data Processing at ID31
==============================

Workflows can be triggered automatically for every scan that has a diffraction camera that needs XRPD data reduction.

Usage
-----

The automatic XRPD data reduction workflows can be enabled through the *xrpd_processor* object

.. code-block:: python

    SIXC [1]: xrpd_processor.enable(p3)
    SIXC [2]: xrpd_processor.options["pyfai_config"] = "/data/visitor/a311182/bm31/20221002/Si_125.poni"
    SIXC [3]: xrpd_processor.options["retry_timeout"] = 3600
    SIXC [4]: xrpd_processor.options["monitor_name"] = "mondio"
    SIXC [5]: xrpd_processor.options["reference"] = 1
    
    SIXC [6]: xrpd_processor
    Out [6]: XRPD processing:
            enabled               True
            integration_options   {'method': 'csr_ocl_gpu'}
            detector_name         p3
            counter_names         ['mondio', 'detdio']
            pyfai_config          /data/visitor/a311182/bm31/20221002/Si_125.poni
            retry_timeout         3600

The configuration can be a poni file or json file generate manually with the pyfai GUI tools.

Currently all results will be saved in the proposal folder:

.. code-block::

    /data/visitor/<proposal>/bm31/<session>/processed

Disable online data processing

.. code-block:: python

    SIXC [6]: xrpd_processor.disable()
    SIXC [7]: xrpd_processor
    Out [7]: XRPD processing:
            enabled               False
            integration_options   {'method': 'csr_ocl_gpu'}
            detector_name         p3
            counter_names         ['mondio', 'detdio']
            pyfai_config          /data/visitor/a311182/bm31/20221002/Si_125.poni
            retry_timeout         3600
            monitor_name          'mondio'
            reference             1

Installation
------------

Installation on lid31dat1 (data processing)
+++++++++++++++++++++++++++++++++++++++++++

One conda environment called `ewoksworker`

Dependencies installed:

.. code-block:: bash

    pip install ewoksjob[blissworker] ewoksserver[frontend] blissoda ewoksxrpd ewoksid31

Processes registered with the supervisor

- ewoksmonitor: monitor data processing http://lid31dat1:5555/
- ewoksserver: create workflows http://lid31dat1:5000/
- ewoksworker: celery worker that does the actual calculation
- sixcprocessor: listens to the SIXC session and sends workflow to the worker

Installation on bibhelm (acquisition control)
+++++++++++++++++++++++++++++++++++++++++++++

Add ewoks configuration to beacon.

Dependencies installed in the bliss environment

.. code-block:: bash

    pip install blissoda[id31]

Add this to every session setup script that needs online data processing

.. code-block:: python

    from blissoda.id31.xrpd_processor import XrpdProcessor as _XrpdProcessor
    xrpd_processor = _XrpdProcessor()