name: singlecellprojections-usage description: Usage instructions for SingleCellProjections.jl (SCP), a Julia package for analysis of single-cell expression data. —
SingleCellProjections usage
SingleCellProjections.jl (SCP) analyses single-cell expression data, and can project new samples onto an existing analysis. It builds on ReproducibleJobs.jl: SCP functions don't compute anything, they return a Job, a recipe for what to compute. fetch!(job) computes the result.
Results are memoized (in memory, and on disk for some jobs), so we can express analyses as they should be performed without paying the full computational cost every time. Computations are lazy. There is no need to keep results around as variables in Julia, because previous results can be fetched quickly when needed (keeping them around can waste memory). Only fetch! what is needed: often an end result, or just its annotations (see below).
This skill doesn't cover all of SCP yet. For functions and options not described here, read their docstrings (e.g. @doc SCP.pseudobulk), and inspect results instead of assuming.
Abbreviations
- obs - observations (typically cells)
- var - variables (typically genes)
- ADT - Antibody-Derived Tags (CITE-seq protein measurements)
- DAG - Directed Acyclic Graph
- PCA - Principal Component Analysis
- LOF - Local Outlier Factor
Packages
At startup, load these packages:
- SingleCellProjections.jl (do both
using SingleCellProjectionsandimport SingleCellProjections as SCP) - ReproducibleJobs.jl (
fetch!,print_spec) - CSV.jl (needed by
SCP.load_csv) and DataFrames.jl - UMAP.jl or TSne.jl, only when computing those embeddings
The functions returning Jobs are public but not exported: call them as SCP.pca etc. A few helpers are exported, e.g. twogroup_covariate.
SCP relies heavily on threading, so start Julia with threads (e.g. julia -t auto).
Data matrices
A DataMatrix is an annotated matrix with variables as rows and observations as columns.
data.var/data.obs-DataFrames with variable / observation annotations. The first column is always the ID column, and IDs must be unique.data.matrix- the matrix. Its type depends on the stage of the analysis: sparse counts, a lazy Matrix Expression after transformation/normalization (e.g.A+B₁B₂B₃), and a denseMatrix{Float64}after PCA or embedding. It's rarely needed directly.
A DataMatrix Job is split into three component Jobs, so we can work with one part without computing the others:
SCP.get_obs(job)/SCP.get_var(job)- just the annotations. Much quicker to fetch than the wholeDataMatrix.SCP.get_matrix(job)- just the matrix.SCP.nobs(job)/SCP.nvar(job)- the number of observations / variables.
Fetched results are shared between Jobs: never mutate them. Use copy before changing a fetched DataFrame.
Loading data
SCP.load_counts loads 10x .h5 files (or CellRanger .mtx[.gz] files):
raw_counts = SCP.load_counts(paths; sample_names)- Cell IDs are
"$(sample_name)_$(barcode)", and thesample_nameobs column holds the sample. - By default only Gene Expression features are kept. For ADT, use
prefilter="feature_type"=>isequal("Antibody Capture").
To pick and load samples from one or more datasets, see sample-tables.md.
Annotations are tables whose first column holds the IDs. Load them with SCP.load_csv(path) (CSV or TSV) and add them with SCP.annotate_obs(data, annots).
Standard pipeline
Counts are QC-annotated and filtered, transformed, normalized (centered, regressing out covariates such as fraction_mt), reduced by PCA, and embedded in 2 or 3 dimensions for visualization:
# QC annotations
counts = SCP.var_counts_fraction(raw_counts, "fraction_mt", "name"=>startswith("MT-"))
counts = SCP.var_counts_sum(counts, "total_RNA_count")
counts = SCP.var_counts_sum(!iszero, counts, "nonzero_RNA_count")
counts = SCP.obs_counts_sum(!iszero, counts, "nonzero_cell_count")
# Quality filtering
filtered = SCP.filter_obs("fraction_mt" => <(0.15), counts)
filtered = SCP.filter_obs("total_RNA_count" => >(1000), filtered)
filtered = SCP.filter_obs("nonzero_RNA_count" => >(500), filtered)
transformed = SCP.sctransform(filtered)
normalized = SCP.normalize_matrix(transformed, "fraction_mt")
reduced = SCP.pca(normalized; nsv=40)
fl = SCP.force_layout(reduced; ndim=3, seed=4567, k=100, k_projection=25)The parameter values are starting points; suitable values depend on the dataset.
Filtering -
SCP.filter_obs/SCP.filter_vartake acolumn => predicatepair, or an annotation table with a predicate, e.g.SCP.filter_obs(SCP.annotation(annots, "doublet_score") => <=(0.65), counts). Cells missing from the table getmissing, which the predicate must handle (e.g.<=(0.65) ∘ Base.Fix2(coalesce, 0.0)).Transformation -
SCP.sctransformis the default choice.SCP.logtransform(log(1 + x * scale_factor / total_counts)) keeps the matrix sparse, which the Mann-Whitney test needs.Normalization - always normalize before PCA; at the very least the data must be centered. More obs annotations (numerical or categorical) can be regressed out.
PCA -
nsvis the number of components, typically 40-100.Embeddings -
SCP.force_layout,SCP.umap(needs UMAP.jl) andSCP.tsne(needs TSne.jl), all withndim2 or 3. Rotate an embedding so that chosen cell groups point along the axes (the first group up):transform = SCP.find_optimal_coord_transform(fl, "celltype"=>isequal("HSC"), "celltype"=>isequal("T-cells"), "celltype"=>isequal("B-cells")) fl = SCP.transform_coords(fl, transform; keep_var=true)
Analyses should be performed on the normalized or PCA-reduced data, not on embeddings. Embeddings are for visualization. Statistical tests are the exception: they take transformed data that isn't normalized (see statistical-tests.md).
Projections
SCP.project applies an analysis built on base data to new data. It takes a Job and one or more old => new replacements, and repeats every step between them with the models fitted on the base data (transformation parameters, normalization, PCA, embedding):
proj_raw_counts = SCP.load_counts(proj_paths; sample_names=proj_names)
proj_fl = SCP.project(fl, raw_counts=>proj_raw_counts)The result contains only the projected observations; plot the base data as background.
Variable IDs must match between the base and the new data (same features).
Any Job built on the base data can be projected, e.g. tests or outlier scores, not only embeddings.
Give one pair per input that differs, e.g. both RNA and ADT counts.
Only the steps after the replaced Job are repeated. E.g. to skip the quality filtering for the projected cells, replace the filtered Job:
proj_counts = SCP.project(counts, raw_counts=>proj_raw_counts) proj_reduced = SCP.project(reduced, filtered=>proj_counts)
Annotation tables
Annotation tables are tables (DataFrames, or Jobs returning them) whose first column is the ID.
SCP.annotation(table, col)- a two-column table with the ID column andcol. Commonly passed to filtering, covariates and tests.SCP.get_columns(table, cols...)- a table with the given columns.SCP.value_column_data(table)/SCP.id_column_data(table)/SCP.column_data(table, col)- a column as a vector.SCP.annotate_obs(data, table)/SCP.annotate_var(data, table)- left-join a table ontoobs/var.SCP.add_obs_column(data, name, values)- add a column whose values are inobsorder.SCP.transform_annotation(f, table)- applyfto the values of a two-column table.SCP.table_leftjoin(a, b)/SCP.create_table(col => values, ...)- join or create tables.
Get an obs annotation as a vector, in the same order as the observations:
celltype = fetch!(SCP.value_column_data(SCP.annotation(SCP.get_obs(data), "celltype")))Inspecting
fetch!(SCP.get_var(transformed))- the variables (e.g. gene names) left after transformation.fetch!(SCP.get_obs(data))- the observation annotations.print_spec(job; [maxdepth])- print a Job as a DAG (shown as a tree with references). Output can be redirected to an IO object.
Other analyses
- Statistical tests - see statistical-tests.md.
SCP.pseudobulk(normalized, "sample_name")- aggregate cells into one observation per sample (more covariates give one observation per combination).SCP.population_matrix(SCP.get_obs(data), "sample_name"; new_var_covariates="celltype")- the fraction of cells of each cell type, per sample.SCP.local_outlier_factor(reduced, normalized; k=10)- an LOF score per cell: how much of an outlier the cell is compared to its neighbors. When projected, only neighbors in the base data are considered, so it shows cells unlike any in the base data.SCP.signature(transformed, "name"=>in(genes), "my_signature")- a signature score per cell, the first principal component of the given genes after normalization.SCP.transfer_annotation(reduced, proj_reduced, "celltype"; k=10)- transfer an obs annotation of the base data to the projected cells by kNN voting, with a confidence score.
Plotting
SCP has no plotting functions; use any plotting library. Fetch the coordinates and an annotation vector in the same order as the observations:
coords = fetch!(SCP.get_matrix(fl)) # ndim × nobs Matrix
celltype = fetch!(SCP.value_column_data(SCP.annotation(SCP.get_obs(fl), "celltype")))For projections, plot the base data's coordinates as background.