Sparse Multidimensional Iterative Lineshape Enhanced NUS Reconstruction

From the Bax Group at the National Institutes of Health ... SMILE: Sparse Multidimensional Iterative Lineshape Enhanced

SMILE NUS Manual

PDF version (very old; see the updates below for the changes that have been made since the manual was written in November 2015.)
HTML version under construction

Citing SMILE

J. Ying, F. Delaglio, D.A. Torchia, and A. Bax: Sparse Multidimensional Iterative Lineshape-Enhanced (SMILE) Reconstruction of Both Non-Uniformly Sampled and Conventional NMR Data, J Biomol NMR 68, 101-118 (2017). (main text, 1,823,907 bytes) (Supplementary Materials, 539,888 bytes)

Installing SMILE

SMILE must be installed along with NMRPipe, and requires the latest version of NMRPipe posted March 15 2018 or later from the new NMRPipe installation site.

If the plugin.smile.tZ file is included with the other .tZ NMRPipe installation files, SMILE will be installed automatically during the NMRPipe installation.

SMILE Plugin Download:	https://spin.niddk.nih.gov/bax-apps/software/SMILE/plugin.smile.tZ (77,257,915 bytes, Dec 5, 2019)
New NMRPipe Site:	www.ibbr.umd.edu/nmrpipe/install.html

If users are updating from an earlier version of SMILE, and already have a suitable version of NMRPipe, they can update SMILE by unpacking the plugin file in the NMRPipe installation directory. For example, on Linux:

   cd $NMRBASE
   /bin/rm -f plugin.smile.tZ
   wget https://spin.niddk.nih.gov/bax-apps/software/SMILE/plugin.smile.tZ
   tar zxvf plugin.smile.tZ

And on Mac OS X:

   cd $NMRBASE
   /bin/rm -f plugin.smile.tZ
   curl -O https://spin.niddk.nih.gov/bax-apps/software/SMILE/plugin.smile.tZ
   cat plugin.smile.tZ | zcat | tar xvf -

Depending on your OS type (linux9, linux212_64, mac, mac11, mac11_64), three environment variables (NMR_PLUGIN_FN, NMR_PLUGIN_EXE, and NMR_PLUGIN_INFO) defined in the coresponding nmrInit.{osType}.com under the $NMRBASE/com directory need to be updated so that the SMILE and FLATT functions are to be included into NMRPipe. If the SMILE plugin contained in the executable nusPipe is the only user plugin installed or to be installed, you can run the updateInit.com script provided in this release to update the settings of these variables. Otherwise, a manual update is recommended. To manually update, just append the content in the files under $NMRBASE/smile onto the environment variable that must have been already defined in the $NMRBASE/com/nmrInit*.com file(s).

SMILE NUS Updates

Version 2.1 Revision 2019.337.11.19 released on December 5, 2019 is available for download. For all the latest features to work properly, it requires a version of NMRPipe posted March 15 2018 or later from the new NMRPipe installation site.

This version is compatible with the past scripts used for the previous releases.
It fixes a bug that may crash the reconstruction on certain linux clusters using Portable Batch System (PBS).
The FLATT algorithm (Guntert and Wurthrich, JMR, 1992, 96, 403-407) is included in this release. Run "nmrPipe -fn FLATT -help" and see Example 12 below for its usage.
This version allows multiple interleaved data sets (e.g. relaxation, CEST, DEST, and ARTSY etc.) to be processed simultaneously as one pseudo nD data set using the -pseudoND flag. See Example 12 for how -pseudoND is used to co-process the attenuated ARTSY with the reference spectrum that has a much better intrinsic S/N.

Version 2.0 beta Revision 2018.094.15.20 released on April 5, 2018.

This version can reconstruct 2D-4D RQD (random quadrature detection) NUS data. For a detailed discussion about RQD, refer to the JBNMR paper from Bostock, Holland, and Nietlispach and the references therein.
The new version is compatible with the past scripts used for the conventional NUS reconstruction. These scripts should run just fine without any changes.
We recommend a new format (two columns per indirect dimension with a header) for the RQD sampling schedule. A temporary python script (rqdSample.py) for generating a 2D-4D RQD sampling schedule is included in this release. Use "rqdSample.py --help" to print brief instructions for this script.
Refer to the 3D noesy-hsqc data collected using RQD in Example 11 below for the pulse programming, Bruker RQD data expansion before conversion, and the RQD reconstruction script.
The current version of SMILE automatically determines whether a data set is collected using RQD. No special command line argument flag is needed, except that if an indirect dimension is recorded using echo-antiecho, users must specify -EA for that dimension (e.g. -xEA, -yEA, or -zEA, corresponding to the -y, -z, or -a axis in the bruk2pipe conversion script). Currently, SMILE only supports one indirect dimension to use the echo-antiecho mode, but it can be any indirect dimension.
Two temporary TCL scripts (rqdExpand.tcl and rqdCompress.tcl) are included. They are the RQD version of the nusExpand.tcl and nusCompress.tcl in the current nmrPipe release, but only work for Bruker ser data. These scripts eventually will be replaced by more general TCL scripts in a later nmrPipe release. Use "rqdExpand.tcl -help" or "rqdCompress.tcl -help" to print brief instructions for the scripts.
The following new features and options are provided that may be useful:

-sampleAverage: averages the duplicate samples in a conventional or RQD sampling schedule;

-maxTime: specify the maximum wall time available for the SMILE calculation; note that the time for the subsequent nmrPipe processing after the reconstruction is not included. This is useful for cluster users who usually have a limited wall time;

smile.stop: a file that users can create in the directory where the SMILE script is started; it instructs SMILE to exit immediately after the current iteration completes; when SMILE is terminated this way, it exits without losing the data already reconstructed at this point.

Version 1.1 Revision 2017.153.15.06 released on June 2, 2017.

This version eliminates the need for users to set -nDim, -nThread, and -maxMem. SMILE obtains the number of dimensions from the header, and sets the number of threads to half of the number of available logical CPUs (or 1, whichever larger). The maximally allowed memory for SMILE (-maxMem) is automatically set to 80% of the installed physical memory. These SMILE options are still available if users prefer to use a different value than the default.
When a SMILE job is submitted via the SLURM scheduler to a Linux cluster, the program sets -nThread automatically to the value defined by the SLURM environmental variable (SLURM_CPUS_PER_TASK). If this variable is undefined, SMILE then sets -nThread to half of the number of the available logical CPU on a node. SMILE sets -maxMem automatically to 80% of the memory defined by SLURM_MEM_PER_NODE, and if this is not defined, maxMem is automatically set to 80% of SLURM_MEM_PER_CPU * SLURM_CPUS_PER_TASK.
There is no output when users set "-report 0". However, a detailed information about the data and the reconstruction is printed if the report level is set to 1. At the report level 2, an additional brief summary about the number of signals, the total iteration time as well as the time for FT is printed at certain iteration numbers (5 for 4D, 10 for 3D, and 20 for 2D). The timing information can be helpful for doing a short test to determine how much computational resource should be used and if the reconstruction should be started with thread binding discussed below. SMILE also outputs the detailed data and reconstruction information in a log file (default: smile.log, use the -log option to change).
The default value for -maxIter is changed to 800, and the default apodization window is now a cosine squared function (i.e. -xQ3 ... is set to 2 instead of 1).
This version also fixes a bug that can crash the program when -sampleCount is smaller than the total number of samples in a schedule, and when the sample at sampleCount+1 is larger than any of the first sampleCount samples.
This version no longer requires the direct dimension to have an even number of points. Therefore, there is no need to set "-round 2" for the EXT function when extracting the region of interest during the processing of the direct dimension.

Version 1.0 Revision 2016.309.13.00 was released on November 4, 2016.

This version changes the default value for the minimum linewidth that SMILE can use during the reconstruction. It also changes the way how the experimentally sampled data are apodized to avoid the potential small truncation wiggles.

Version 1.0 Revision 2016.245.17.30 was released on September 1, 2016.

This release includes a new build compiled using the Intel's C compiler and Math Kernal Library on 64-bit Mac OS X El Capitan (10.11.1), which should run much faster than the previous program. However, the program may require Mac OS X Yosemite or higher to run properly. Also the 32-bit Mac version is no longer supported, but can be made available on request.
The 32- and 64-bit Linux versions are updated too.
Major changes in this version:

By default, all indirect dimensions are automatically extended by 50% during the reconstruction to improve the spectral resolution unless a user explicitly instructs SMILE not to make such an extension. The final output from SMILE is therefore 1.5x1.5x1.5=3.375 times larger than the input data size for 4D. The extension also increases the minimum memory requirements by SMILE.
SMILE now can directly extend the indirect dimensions of fully sampled data without using nusZF.com to first expand the data and to generate a sequential sampling list. For fully sampling data, SMILE internally creates a sampling schedule and users no longer need to provide one. This application allows SMILE to be used as a convenient alternative to Linear Prediction.
SMILE downscales the reconstructed signals before adding them back to the residuals. The scaling factor is now estimated from an on-resonance simulated signal with a decay rate calculated from the average linewidth of the peaks selected in the first iteartion. The new verion of SMILE also allows users to provide a scaling factor, which is particularly useful for spectral reconstruction of NMR relaxation data. Use of an identical scaling factor prevents the intensity variation due to slightly different scaling from one relaxation time point to another.

Beta test version released on March 22, 2016

A new Linux version of SMILE is availabe. The new Mac version has yet to be built.
Once the new executable is downloaded, you may want to rename the old nusPipe program to nusPipe.20151119 or whatever you prefer before moving the new program to the same directory. After the new nusPipe executable file is placed in the correct nmrbin directory, you may need to change its permissions.
The new version is optimized for a better performance mainly by using the Intel Math Kernal Library (MKL). The actual spectral reconstruction algorithm remains unchanged. Therefore, the quality of the reconstruction should be very similar but the new program will run much faster. The 3D NOESY-HSQC reconstruction in Example 4 below completed in ~16 min (previously 1.3 hours) and the 4D HMQC-NOESY-HMQC in Example 5 took 1.5 hours (compared to previous 5.6 hours).
Changes in this version:

The executable file is much larger (~30 MB). This is because more Intel libraries are statically linked so that users without the commercial libraries can still run the program.
The command line argument "-report" can now take 0 (no smile.log saved at the end of the reconstruction), 1 (save critical reconstruction parameters in smile.log), or 2 (output the time each iteration takes and the time spent on the slowest steps such as FT, signal reconstruction, and array initialization, in addition to saving the SMILE parameters in smile.log).
The input data SMILE reads from the pipe is now compressed and all the unsampled points filled with zeros during the NUS expansion are removed. This lowers the minimum memory requirement from previous ~2.5 to ~1.5 times the total input data. For example, as described in the manual above, the input data of the 4D HMQC-NOESY-HMQC in Example 5 below is around 6.9 GB and the previous version of SMILE requires ~17 GB of memory. SMILE now only needs ~9.8 GB to run. However, note that this requirement does not include the memory space that the Intel libraries may need during the reconstruction. It is not clear exactly how much memory these libraries need, but the requirement depends on the data size. When SMILE was given 9.8 GB (i.e. -maxMem 9.8), the reconstruction ran fine on a workstation that has totally 12 GB memory. However, when 100 GB was provided to SMILE (i.e. -maxMem 100) on a computer with 124 GB total memory, the reconstruction actually used around 15 GB extra memory space, i.e. the total memory used was ~115 GB. To avoid using the swap memory space, users are advised to set -maxMem to be ~20% smaller than the total memory available.
Depending on the hardware topology of user's computers, binding the openMP threads to the physical CPUs throughout the reconstruction by setting the KMP_AFFINITY environment variable properly in the processing script before calling the SMILE function may further improve the performance of the new program. For example, binding of 64 omp threads to 64 AMD cores without hyperthreading can be done using one of the two equivalent settings:
```
setenv KMP_AFFINITY "verbose,granularity=fine,compact,0,0" or explicitly:
setenv KMP_AFFINITY "verbose,granularity=fine,proclist=[0-63],explicit"
```
in a csh shell, or
```
export KMP_AFFINITY="verbose,granularity=fine,compact,0,0" or explicitly:
export KMP_AFFINITY="verbose,granularity=fine,proclist=[0-63],explicit"
```
in a bash shell.

For CPUs capable of hyperthreading, one thread per core is still recommended for SMILE, which can be achieved by setting the permute value to 1. For example, binding 32 threads to 32 physical cores capable of running 64 threads, either of the following settings allows users to run one thread per core without using hyperthreading:
```
setenv KMP_AFFINITY "verbose,granularity=fine,compact,1,0" or explicitly:
setenv KMP_AFFINITY "verbose,granularity=fine,proclist=[0-63:2],explicit"
```
in a csh shell, or
```
export KMP_AFFINITY="verbose,granularity=fine,compact,1,0" or explicitly:
export KMP_AFFINITY="verbose,granularity=fine,proclist=[0-63:2],explicit"
```
in a bash shell.

See Intel thread affinity interface for more details. To unset the KMP_AFFINITY environment variable, users can use "unsetenv KMP_AFFINITY" in a csh shell and "unset KMP_AFFINITY" in a bash shell. The thread affinity setting is not required to run SMILE, but may help the program run faster. To find out whether the thread binding helps, users can run a few iterations of SMILE with and without the binding (using "-report 2" in both runs). Choose the setting that gives the best performance.

SMILE NUS Examples (updated on April 5, 2018)

Example 1: 10 simulated peaks on a 2D slice with 10% NUS sampling (file size: 6,752 bytes), simulated and reconstructed data (file size: 21,382,178 bytes).
Example 2: 20% NUS reconstruction of 2D TROSY (file size: 66,585 bytes), fully sampled and pseudo NUS sampled Bruker ser files (file size: 74,224,221 bytes), and conventionally processed and SMILE reconstructed data (file size: 129,132,136 bytes).
Example 3: SMILE as an alternative to LP for extending the constant-time acquisition from 28 to 56 ms in a fully sampled CT-[13C-1H]-HSQC experiment (file size: 52,496 bytes), Bruker ser file (file size: 2367856 bytes), and conventionally processed and SMILE extended data (file size: 21,477,209 bytes).
Example 4: 5% NUS reconstruction of 3D HNCO (file size: 96,396 bytes), Bruker ser file (file size: 138,247,993 bytes), and SMILE reconstructed data (file size: 2,791,274,003 bytes).
Example 5: 30% NUS reconstruction of 3D 13C NOESY-HSQC (file size: 141,706 bytes), Bruker ser file (file size: 224,907,966 bytes), and SMILE reconstructed data using different apodization functions for the CT 13C dimension (file size: 12,213,211,027 bytes).
Example 6: 2.58% NUS reconstruction of 3D (H)N(COCO)NH for measurement of 3J(C'C') couplings (file size: 28,385 bytes), fully sampled and pseudo NUS sampled Bruker ser files (file size: 457,046,105 bytes), and SMILE reconstructed data (file size: 1,869,183,016 bytes).
Example 7: 1.56% NUS reconstruction of 4D methyl HMQC-NOESY-HMQC (file size: 76,746 bytes), Bruker ser file (file size: 160,138,996 bytes), and SMILE reconstructed data (file size: 23,957,612,020 bytes). Note that the final 4D reconstructed spectrum (ft/test%03d.ft4) is more than 300 GB large and not included here. Users can generate the spectrum from ft1.smile/test%03d.ft1.
Example 8: Accuracy and performance comparison of different NUS sampling schemes (random, T2 weighted, Poisson gap without weight or with a sinusoidal weight) (file size: 10,670,558 bytes), simulated and reconstructed data (file size: 22,743,250,708 bytes).
Example 9: 3D 15N NOESY-HSQC peak intensity comparison between the fully sampled spectrum and the NUS spectra reconstructed by SMILE from a fraction (8-15%) of the fully sampled data:

For 350uM ABeta at 4 C: acquisition parameters, processing and reconstruction scripts (file size: 269,756 bytes), Bruker ser file, processed data, and reconstructed spectra (file size: 5,093,533,238 bytes). Here is the intensity correlation plots for the diagonal and cross peaks at different sparsity:

For ubiquitin at -5 C under high pressure: acquisition parameters, processing and reconstruction scripts (file size: 421,084 bytes), Bruker ser file, processed data, and reconstructed spectra (file size: 4,609,938,642 bytes). Here is the intensity correlation plots for the strong and weak peaks at 15% sparsity:
Example 10: Using SMILE to repair bad FIDs in a uniformly sampled 3D 15N noesy-hsqc spectrum (file size: 3,651,943 bytes), Bruker ser file and reconstructed spectrum (file size: 664,052,121 bytes).
Example 11: 20% 3D RQD NUS NOESY-HSQC recorded with echo-antiecho (file size: 118,121 bytes), Bruker ser file and SMILE reconstructed spectrum (file size: 4,693,668,099 bytes).
Example 12: 15% interleaved 3D N-N-H ARTSY baseline corrected using FLATT and reconstructed using the new pseudo nD approach (file size: 373,186 bytes), Bruker ser file, baseline test spectrum, and SMILE reconstructed pseodu 4D spectrum (file size: 2,126,993,631 bytes).

Contacts

[ Home ] [ NIH ] [ NIDDK ] [ Terms of Use ]
last updated: April 5, 2018 / Jinfa Ying