Data export

From SpinozaWiki
Jump to: navigation, search


Introduction

Imaging data acquired on the MRI scanners at the Spinoza Centre are automatically transferred and stored on the file server in the DICOM format. The data can subsequently be accessed (from within and outside the Centre) using the XNAT platform. Within XNAT, data can e.g. be converted to Nifti, and analysis pipelines can be created and automatically run on the data. Other (imaging) data can be dropped in the project folder accessible via SFTP. These options make it unncessary to attach local storage such as hard drives or USB sticks to the host computer, which therefore is no longer allowed. In the following, the process from data acquisition to data access is described.

Start of a project

For new studies, a project is defined in XNAT. A user account wll be created for the primary researcher as indicated on the project form. Additional accounts can be requested from spinoza-support@spinozacentre.nl. In addition, a project folder is created on the ftp server. From the scanners, files as well as folders can be dragged into this folder (e.g. physiology log files). The SC staff will complete the required steps to automate date export.

During and after scanning

Scans are immediately forwarded to the network after completion (i.e. reconstruction is finished). Depending on the amount of data in a scan, this will take more or less time.

Xnat archived.png
After a session is fully processed by XNAT, you will receive an email with a link to review the content of the archived data (see example on the right).

If you do not receive an email within roughly an hour after scanning, send an email to scanning@spinozacentre.nl so SC staff can investigate the cause.


Wdcmproc.png
In addition to the email you receive from XNAT it is possible to receive additional emails detailing the export process (see example) allowing you to follow the export in more detail. This option is disabled by default.


XNAT

Paraphrasing the XNAT website, XNAT is an open source imaging informatics platform. It facilitates common management, productivity, and quality assurance tasks for imaging and associated data. Thanks to its extensibility, XNAT can be used to support a wide range of imaging-based projects [1]. The Spinoza XNAT system can be found at https://ftpsc.spinozacentre.nl/xnat/.

  • Upon login, you will see the home screen where you will see an overview of the projects you are involved with (either as a member or owner), plus the projects that have a public or protected status (by default, your project will be defined as private). See the images below for details.
  • Clicking a project will show a list of MR sessions that have been stored. In addition, several tabs are available:
  • Access: for controlling who can see the project and/or access data
  • Manage: for setting, among other things, who should receive email from XNAT (e.g. on incoming data), and what should be done with incoming data that conflicts with existing data
  • Pipelines: for enabling/disabling the automatic execution of (analysis) scripts on data, e.g. DICOM to NIFTI conversion.
  • Clicking on a session will show an overview of the data stored in XNAT with details such as file size and acquisition series number. Clicking the '+' gives additional info on the acquired sequence. Hovering over the details in the "File" column will show additional data types (e.g. if images have been converted to NIFTI).

Downloading data

Data can be downloaded from XNAT through two different interfaces or through a set of Matlab scripts.

Manage files

Manage files.png

This interface allows you to download files (dicoms, nifti’s generated in the dcm2nii pipeline, quality assessment measures from the QAP pipeline) as a single compressed file (zip, tar or tar.gz). When this function is selected after opening a scan session, a window is opened showing all files for that session. You can easily (de)select the files of your choosing and press download.

Download images

Download images.png

The main download interface is currently Java-based, and allows for downloading of data at the project and session level. Regretfully, this interface has very limited browser support. You can use either Internet Explorer, Safari or the "Extended support release" of Firefox.

In the download window, you can

  1. select which sessions should be included in the download (unless you are downloading on session level, in which case only the specific session will be available)
  2. select which data formats (e.g. DICOM, NIFTI) and sequences (EPI, T1 etc.) should be exported
  3. control (to a certain extent) the structure of the downloaded files by moving separate subjects into subfolders

It is also possible to download data using File:MatNat.zip, a set of Matlab scripts.

Pipelines

Several automated operations/analyses can be performed on the data when they are stored in XNAT. The most important ones currently in place are

  • DICOM to NIFTI conversion (based on dcm2niix)
  • Quality Assessment: calculation of functional and structural imaging quality metrics (based on the PCP Quality Assessment Protocol

Pipelines can be added to a project by

  • clicking the "Pipelines" tab in the project view, and subsequently
  • clicking "Add more pipelines",
  • choosing the desired pipeline(s) from the pipeline overview
  • defining the desired settings
    • Note: make sure to choose auto launch when adding the pipeline as this setting cannot be modified later on
    • Note: please do not switch on the 'debug' option in the Pipeline Setup

QA (WIP)

You can use the QAP pipelines to calculate some basic quality metrics of your data. We are currently assessing to what extent the in vivo results obtained by running the PCP Quality Assessment Protocol provide reliable information on the scanner status.

Exporting and accessing non-DICOM data

If you collect other data, e.g. raw data or physiology, these can be exported to the network drive mounted on the scanners. Projects are located in S:\projects. Note that while you cannot enter your project folder (for privacy reasons), you can drag and drop files and folders into them and subsequently access them via FTP.

Some notes on exporting data to local drives:

  • Data stored on E:\Export, G:\Temp etc. should be removed after the files are copied to the project folder.
  • Data stored in these locations will be removed by SC staff without prior notice and is not backed up.
  • Never save data to C:\ or D:\

SFTP

  • The ftp server address is sftp://ftpsc.spinozacentre.nl/.
  • We recommend using FileZilla as client to connect to the FTP server.
  • You can log in with the credentials you received upon starting your project. Note that at the moment, your XNAT and FTP credentials are not linked, of course you can choose the same password for both.

Data deletion schedule

3 Tesla

Note that this schedule should be regarded as a guideline: data is not guaranteed to remain on the system for the indicated duration.
If you encounter problems during or after exporting (e.g. not all data was exported correctly) contact us at scanning@spinozacentre.nl as soon as possible.

Scanning day Deletion day*
Monday Wednesday
Tuesday Thursday
Wednesday Friday
Thursday** Monday
Friday** Monday
Saturday Monday
Sunday Tuesday

* This could be as early as 7.00 in the morning!
** For these days there is an increased risk of data removal before the indicated day, depending on the weekend scanning schedule.

7 Tesla

Due to the large amount of raw data being acquired at the 7 Tesla, it can generally not be guaranteed that data are retained for more than roughly 1 day. If you don't manage to (fully) export your data during your scanning session, or find out afterwards not all data were exported (correctly), get in touch with SC staff asap to reduce the risk of losing data.