Data Matrices

DataMatrix objects — annotated matrices where rows are variables and columns are observations — are central in SingleCellProjections.jl.

Let's load a sample to see what a DataMatrix looks like:

using SingleCellProjections
import SingleCellProjections as SCP
using ReproducibleJobs
using SparseArrays

sample_path = joinpath("samples", "AML28.h5")

"samples/AML28.h5"

counts = SCP.load_counts(sample_path; sample_names="AML28")
c = fetch!(counts)

DataMatrix (32738 variables and 6905 observations)
  Block Matrix (32×7)
  Variables: id, name, feature_type, genome, read, pattern, sequence
  Observations: cell_id, sample_name, barcode

The display shows the matrix size (number of variables and observations), a brief description of the matrix contents, and an overview of available variable and observation annotations.

Variables

Variables, or var for short, are typically genes, features (such as CITE-seq features), or variables after dimension reduction (e.g. "UMAP1"). The variables are stored as a DataFrame and can be accessed by:

c.var[1:6, :]

Observations

Observations, or obs for short, are typically cells, but can also be pseudobulk samples, for example. The observations are stored as a DataFrame and can be accessed by:

c.obs[1:6, :]

IDs

Each variable and each observation must have a unique ID. The first column of the var and obs DataFrames is always the ID column.

Most of the time, IDs are handled automatically by SingleCellProjections.jl. Sometimes, you need to make sure IDs are unique when loading or merging data matrices. In particular, when loading a DataMatrix that should be projected onto another DataMatrix, the user must ensure that variable IDs match.

Matrix

The matrix can be accessed by data.matrix. Depending on the stage of analysis, different kinds of matrices (or matrix-like objects) are used. Most of this complexity is hidden from the user, but internally SingleCellProjections.jl depends on this functionality to be fast and to reduce memory usage.

Roughly, the matrix types used at different stages are:

1. Counts — SparseMatrixCSC (often in a blocked format)
2. Transformed and normalized data — Matrix Expressions
3. PCA result — Matrix{Float64}
4. ForceLayout/UMAP/t-SNE result — Matrix{Float64}

Jobs and DataMatrices

A DataMatrix Job is internally split into three component Jobs: one for the matrix, one for var, and one for obs. You can access these with:

• SCP.get_matrix(job) — the matrix component
• SCP.get_var(job) — the variable annotations
• SCP.get_obs(job) — the observation annotations

Operations that only affect one component leave the others unchanged. For example, SCP.logtransform only transforms the matrix — it passes var and obs through without modification. This means the var and obs Jobs are literally the same object (identical by ===) before and after the transformation:

transformed = SCP.logtransform(counts)

# var is unchanged by logtransform — same Job object
forward!(SCP.get_var(transformed)) === forward!(SCP.get_var(counts))

true

# matrix is different — logtransform created a new matrix Job
forward!(SCP.get_matrix(transformed)) === forward!(SCP.get_matrix(counts))

false

This splitting is what makes the caching and projection system efficient — unchanged components are never recomputed or stored redundantly.