Online Data Processing at ID22
==============================

Usage
-----

The automatic *stscan* workflows can be configured through the *stscan_processor* object

.. code-block:: python

    EXP [1]: stscan_processor
             Out [4]: Workflows:
                   convert                /tmp/demo/convert.json 
                   rebin/sum              /tmp/demo/rebinsum.json
                   extract                /tmp/demo/extract.json 
                  
                  Execution:
                   session_in_outprefix   True
                  
                  Convert:
                   do_convert                 True                                            
                   Results:                                                                   
                    primary                   opid22@diffract22new:/users/opid22/data1/       
                    secondary                 opid22@lid22bliss:/users/opid22/data1/     
                    backup                    opid22@lid22bliss:/data/id22/backup/data22/
                    include_proposal_outdir   False                                           
                  
                  Rebin:
                   do_rebin                   True                                                                   
                   Parameters:                                                                                       
                    range                     [0.0, inf]                                                             
                    delta2theta               [0.003]                                                                
                    startp                    31                                                                     
                    parsfile                  /data/id22/inhouse/CD_GC_PDF/advanced_50keV/patterns/for_wout/out7.pars
                   Results:                                                                                          
                    primary                   opid22@diffract22new:/users/opid22/data1/                              
                    backup                    opid22@lid22bliss:/data/id22/backup/data22/                       
                    include_proposal_outdir   False                                                                  
                  
                  Sum:
                   do_sum                     True                                                                  
                   Parameters:                                                                                      
                    binsize                   [0.002]                                                               
                    resfile                   /data/id22/inhouse/CD_GC_PDF/advanced_50keV/patterns/for_wout/temp.res
                   Results:                                                                                         
                    include_proposal_outdir   True                                                                  
                  
                  Extract:
                   do_extract                 <stscan parameter>                       
                   Parameters:                                                         
                    tth_min                   None                                     
                    tth_max                   None                                     
                    full_tth                  False                                    
                    startp                    31                                       
                    inp_file                  /users/opid22/out7_files/LaB6_all7_os.inp
                    inp_step                  2                                        
                   Results:                                                            
                    include_proposal_outdir   True

The *stscan* macro uses the *stscan_processor* to submit several workflows after every scan.

To submit a workflow manually:

.. code-block:: python

    EXP [2]: stscan_processor.submit_workflows()

Installation
------------

Installation on diffract22new (data processing)
+++++++++++++++++++++++++++++++++++++++++++++++

One conda environment called `ewoksworker`

Dependencies installed:

.. code-block:: bash

    pip install ewoksjob[blissworker]
    pip install multianalyzer@git+https://github.com/kif/multianalyzer.git@main
    pip install git+https://gitlab.esrf.fr/workflow/ewoksapps/ewoksid22.git@main

Processes registered with the supervisor

- ewoksmonitor: monitor data processing http://diffract22new:5555/
- ewoksserver: create workflow http://diffract22new:5000/
- ewoksworker: celery worker that does the actual calculation
- ewokssolo1: celery worker that does the actual calculation
- ewokssolo2: celery worker that does the actual calculation

Installation on lid22bliss (acquisition control)
++++++++++++++++++++++++++++++++++++++++++++++++

Add ewoks configuration to beacon.

Dependencies installed in the bliss environment

.. code-block:: bash

    pip install blissoda[id22]

Add this to every session setup script that needs online data processing

.. code-block:: python

    from blissoda.id22.stscan_processor import StScanProcessor as _StScanProcessor
    stscan_processor = _StScanProcessor()