Online Data Processing at ID22#

Usage#

The automatic stscan workflows can be configured through the stscan_processor object

EXP [1]: stscan_processor
         Out [4]: Workflows:
               convert                /tmp/demo/convert.json
               rebin/sum              /tmp/demo/rebinsum.json
               extract                /tmp/demo/extract.json

              Execution:
               session_in_outprefix   True

              Convert:
               do_convert                 True
               Results:
                primary                   opid22@diffract22new:/users/opid22/data1/
                secondary                 opid22@lid22bliss:/users/opid22/data1/
                backup                    opid22@lid22bliss:/data/id22/backup/data22/
                include_proposal_outdir   False

              Rebin:
               do_rebin                   True
               Parameters:
                range                     [0.0, inf]
                delta2theta               [0.003]
                startp                    31
                parsfile                  /data/id22/inhouse/CD_GC_PDF/advanced_50keV/patterns/for_wout/out7.pars
               Results:
                primary                   opid22@diffract22new:/users/opid22/data1/
                backup                    opid22@lid22bliss:/data/id22/backup/data22/
                include_proposal_outdir   False

              Sum:
               do_sum                     True
               Parameters:
                binsize                   [0.002]
                resfile                   /data/id22/inhouse/CD_GC_PDF/advanced_50keV/patterns/for_wout/temp.res
               Results:
                include_proposal_outdir   True

              Extract:
               do_extract                 <stscan parameter>
               Parameters:
                tth_min                   None
                tth_max                   None
                full_tth                  False
                startp                    31
                inp_file                  /users/opid22/out7_files/LaB6_all7_os.inp
                inp_step                  2
               Results:
                include_proposal_outdir   True

The stscan macro uses the stscan_processor to submit several workflows after every scan.

To submit a workflow manually:

EXP [2]: stscan_processor.submit_workflows()

Installation#

Installation on diffract22new (data processing)#

One conda environment called ewoksworker

Dependencies installed:

pip install ewoksjob[blissworker]
pip install multianalyzer@git+https://github.com/kif/multianalyzer.git@main
pip install git+https://gitlab.esrf.fr/workflow/ewoksapps/ewoksid22.git@main

Processes registered with the supervisor

ewoksmonitor: monitor data processing http://diffract22new:5555/
ewoksserver: create workflow http://diffract22new:5000/
ewoksworker: celery worker that does the actual calculation
ewokssolo1: celery worker that does the actual calculation
ewokssolo2: celery worker that does the actual calculation

Installation on lid22bliss (acquisition control)#

Add ewoks configuration to beacon.

Dependencies installed in the bliss environment

pip install blissoda[id22]

Add this to every session setup script that needs online data processing

from blissoda.id22.stscan_processor import StScanProcessor as _StScanProcessor
stscan_processor = _StScanProcessor()