Test page

From Miniscope
Jump to: navigation, search

Overview

This guide covers how to use the Miniscope Workshop Processing and Analysis pipeline: a step-by-step curriculum for going from a raw Miniscope recording and behavior video to a place-cell analysis. Each stage in the pipeline is an independent tool with its own tutorial notebook. All stages share a single Python environment and a common dataset.

The pipeline covers five sequential stages plus a capstone that ties them together:

Before running any stage: Complete the installation guide first — Guide/Workshop Installation (Mac). All commands in this guide assume your virtual environment is active (you see (.venv) in your Terminal prompt) and that you are in the workshop folder.

Pipeline at a Glance

The pipeline processes two parallel streams — neural and behavioral — and fuses them in the capstone:

Stage Tool Input Output Independent?
0 minisim (optional) (code) Simulated recording Yes — standalone
1 Minian Raw miniscope video Denoised traces, spatial footprints Starts neural stream
2 CaTune / CaDecon (calab) Minian traces (C.zarr) Deconvolved activity (activity.npy) Requires Stage 1
3 eztrack Raw behavior video Position CSV Yes — runs in parallel with Stages 1–2
4 CaMAP (capstone) Minian + calab + eztrack + timestamps Place-cell bundle (.camap) Requires Stages 1–3

Stages 0 and 3 are independent of the neural processing chain and can be run in any order. The capstone (Stage 4) requires the outputs of Stages 1, 2, and 3.

Data Organization

All workshop data lives under data/sessions/<session>/, organized by pipeline stage: 

data/sessions/<session>/
├── raw/          miniscope video, behavior video, neural + behavior timestamps
├── minian_out/   Minian output  (C, C_lp, A, max_proj — zarr stores)
├── deconv_out/   calab output   (activity.npy)
└── eztrack_out/  eztrack output (position CSV)

Two sessions are available:

  • prerecorded — a backup dataset hosted on UCLA Dataverse, always available for download. Use this if your live recording is unavailable.
  • live — recorded during the workshop session. Its DOI is provided by the organizer on the day.

Downloading the data

The dataset is hosted on UCLA Dataverse and downloaded automatically by get_data.py, which reads the file list and checksums directly from the archive — no manual steps required. The script is local-first: it checks what you have already produced locally and only downloads the stages that are missing. You will not lose your own pipeline outputs by running it. 

python scripts/get_data.py                          # prerecorded session, all stages
python scripts/get_data.py --what raw               # raw video and timestamps only
python scripts/get_data.py --what processed         # processed stages only (skip raw)
python scripts/get_data.py --what minian_out        # one specific processed stage
python scripts/get_data.py --session live --doi ... # workshop live recording (DOI from organizer)
python scripts/get_data.py --force                  # re-download even if files exist
Timestamps are not produced by any pipeline tool. The files neural_timestamp.csv and behavior_timestamp.csv come directly from the Miniscope DAQ and are included in the raw download. The capstone requires them to align the neural and behavioral clocks.

Stage 0 — minisim (Optional)

minisim lets you simulate a synthetic Miniscope recording from first principles. It is a teaching tool — it is not required to run the rest of the pipeline — but it builds strong intuition for what Minian is actually doing and why each step matters.

What it covers

Notebook What it teaches
01_anatomy Constructs a synthetic recording stage by stage: place neurons → calcium activity → optics → neuropil → brain motion → sensor noise. Each Minian stage inverts one physics stage.
02_demixing Explains why drawing an ROI and averaging pixels does not recover a cell's calcium signal. Shows what CNMF demixing solves and where it breaks down.
03_metrics Uses exact ground-truth from minisim to grade how well the pipeline recovered the true signal. Shows what each metric measures and how to read a recovery report.
build_recording Data-generation studio. Tune optics, tissue, neural activity, and motion; generate a simulated recording in zarr or AVI format with ground truth included.

How to open the notebooks

The teaching notebooks are fetched from the minisim package and placed in tutorials/minisim/notebooks/ by the setup script. Open Jupyter Lab and navigate there: 

jupyter lab

Navigate to tutorials/minisim/notebooks/ in the left panel. Open any notebook and select the Workshop kernel.

Key concepts

  • A Miniscope recording is a physical chain. Numerical aperture, magnification, pixel pitch, scattering depth, and sensor noise set limits that no processing algorithm can recover beyond.
  • The CNMF equation is: movie = A · C + b · f, where A is the matrix of spatial footprints, C is the matrix of calcium traces, and b · f is the neuropil background. Minian solves this.
  • Deconvolved activity S is a scaled activity rate — not a spike train. It should not be binarized when scoring.

Stage 1 — Minian: CNMF Processing

Minian extracts denoised calcium traces and spatial footprints from raw miniscope video using constrained non-negative matrix factorization (CNMF).

Workshop-specific modification: This workshop uses a modified Minian pipeline (pipeline_no_deconv.ipynb) that replaces Minian's built-in temporal deconvolution with a low-pass filter. Deconvolution is handled separately by calab in Stage 2. Do not use Minian's standard pipeline.ipynb for this workshop.

Inputs and outputs

Path Description
Input data/sessions/<session>/raw/ Raw miniscope video (0.avi, 1.avi, ...)
Output data/sessions/<session>/minian_out/C.zarr Raw extracted fluorescence (un-deconvolved) — this feeds calab
Output data/sessions/<session>/minian_out/C_lp.zarr Low-pass-denoised fluorescence — for display and QC
Output data/sessions/<session>/minian_out/A.zarr Spatial footprints (one per cell)
Output data/sessions/<session>/minian_out/max_proj.zarr Maximum projection image of the recording

How to run

Open Jupyter Lab and navigate to tutorials/minian/notebooks/pipeline_no_deconv.ipynb. Select the Workshop kernel and run the notebook cell by cell. Each cell is annotated with what it does and what to look for.

What healthy output looks like: Spatial footprints (A) should show compact, roughly circular cell bodies. The C traces should show individual transients — brief rises followed by exponential decay. If traces are flat, noisy throughout, or show constant ramp-up, something went wrong upstream (check FOV, LED power, and expression quality).

What each Minian stage does

Stage What it inverts
Motion correction Removes brain motion; each Minian stage inverts one physical effect from image formation
Background removal Separates neuropil and leakage from cell-body signal
Spatial initialization Seeds initial estimates of cell locations and footprint shapes
Spatial update Refines footprint shapes given current trace estimates
Temporal update (modified) Extracts C (raw fluorescence); in this workshop, replaces deconvolution with a low-pass filter to produce C_lp
Merge / prune Removes duplicate components; merges spatially overlapping components that represent the same cell

Stage 2 — CaTune / CaDecon: Deconvolution

Deconvolution converts Minian's denoised calcium fluorescence traces into estimates of underlying neural activity. This workshop uses calab — a package containing two tools: CaDecon (automated) and CaTune (interactive parameter tuning). Both open a browser-based interface.

Inputs and outputs

Path Description
Input data/sessions/<session>/minian_out/C.zarr Minian raw fluorescence traces
Output data/sessions/<session>/deconv_out/activity.npy Deconvolved activity, shape (n_cells, n_frames)

Recommended: use the notebook

The primary interface for this stage is tutorials/deconvolution/deconvolution.ipynb. Open it in Jupyter Lab with the Workshop kernel. It walks through both CaDecon and CaTune in one place, with an explore path (hosted browser app, no data needed) and a your-data path (sends your Minian traces to the app and saves activity.npy). This is the recommended approach for the workshop.

Alternative: use the scripts

If you prefer to run from Terminal rather than Jupyter, the same tools are available as scripts:

Option A — CaDecon (automated): 

python tutorials/deconvolution/run_cadecon.py --session prerecorded --fs 20

Option B — CaTune (interactive, two phases): 

python tutorials/deconvolution/run_catune.py --session prerecorded --fs 20

Tune parameters in the browser, export the JSON file, then apply: 

python tutorials/deconvolution/run_catune.py --session prerecorded --params /path/to/exported_params.json

Try without data (demo mode): 

python tutorials/deconvolution/run_cadecon.py --demo
python tutorials/deconvolution/run_catune.py --demo

Which tool to use?

Situation Recommended tool
First time, want to see results quickly CaDecon (automated)
Results look poor or you want to inspect individual cells CaTune (interactive)
No data available, just learning Either — both have a demo mode with simulated data in the browser
What to look for: Good deconvolution produces a sparse activity signal — most values near zero, with brief sharp peaks coinciding with the rising phase of calcium transients. If activity is everywhere or nowhere, the parameters need adjustment.

Stage 3 — eztrack: Behavioral Tracking

eztrack extracts the animal's position from behavior video. It runs independently of the neural processing chain and can be completed at any point before the capstone.

Inputs and outputs

Path Description
Input data/sessions/<session>/raw/behavior.mp4 Raw behavior video
Output data/sessions/<session>/eztrack_out/ Position CSV with columns: frame, x, y, detected, distance_px, ...

How to run

Open Jupyter Lab and navigate to tutorials/eztrack/notebooks/. Two notebooks are available:

Notebook Use when
LocationTracking_Individual Tracking a single behavior video (standard case)
LocationTracking_BatchProcess Processing multiple videos at once

Select the Workshop kernel and follow the notebook instructions. The tracking is LED-based — the notebook guides you through setting the brightness threshold to isolate the tracking LED from the background.

Format note

eztrack outputs a flat CSV. The capstone (CaMAP) expects DeepLabCut-style CSV format (a 3-row header: scorer / bodyparts / coords). The capstone notebook handles this conversion automatically using the eztrack_to_camap() glue function — you do not need to do anything extra.

Stage 4 — CaMAP Capstone: Place-Cell Analysis

The capstone notebook ties together all previous stages. It loads Minian traces, deconvolved activity, eztrack position data, and hardware timestamps; converts formats; aligns the neural and behavioral clocks; and performs place-cell analysis.

Complete Stages 1, 2, and 3 before running the capstone. If you do not have your own outputs from any stage, run python scripts/get_data.py --what processed to download the prerecorded processed data from UCLA Dataverse and use that instead.

Inputs required

☐ File Produced by
☐ minian_out/ (zarr stores) Stage 1 (Minian) or UCLA Dataverse download
☐ deconv_out/activity.npy Stage 2 (calab) or UCLA Dataverse download
☐ eztrack_out/ (position CSV) Stage 3 (eztrack) or UCLA Dataverse download
☐ raw/neural_timestamp.csv Miniscope DAQ — included in raw download
☐ raw/behavior_timestamp.csv Miniscope DAQ — included in raw download

How to run

Open capstone/camap_placecells.ipynb in Jupyter Lab. Select the Workshop kernel. Run the notebook section by section — each section includes inspection plots so you can verify each step before continuing.

What the capstone does, section by section

# Section What it does
1 Setup and data paths Imports glue functions; checks all input files are present; sets session and config paths
2 eztrack → CaMAP format Converts flat eztrack CSV to DeepLabCut-style format using eztrack_to_camap()
3 Timestamps → CaMAP format Converts Miniscope DAQ timestamp CSVs (ms since recording start) to the seconds-based format CaMAP expects, using miniscope_timestamps_to_camap()
4 Load data Loads Minian traces, spatial footprints, max projection, and behavior trajectory
5 Preprocess behavior Applies perspective correction, removes position outliers (Hampel filter), clips trajectory to arena, converts pixels to millimeters; displays trajectory at each cleaning step
6 Inject deconvolved activity Loads activity.npy from calab; aligns rows with Minian unit IDs; bypasses CaMAP's built-in OASIS deconvolution
7 Align two clocks Neural and behavior timestamps are from independent clocks. match_events() linearly interpolates position onto the neural timestamp axis. The result is one row per neural frame with x, y, speed, and all unit activities
8 Inspect timestamps Plots frame index vs. Unix time for each stream — a straight line means healthy clock; dropouts or jumps are visible immediately
9 Compute occupancy Bins the arena into a spatial grid; computes time spent in each bin; marks bins with insufficient occupancy as invalid for rate-map computation
10 Place-cell analysis Runs four tests on every unit (see below)
11 Deep dive on one unit Steps through all four tests manually for a single cell; shows rate map, SI, shuffle distribution, and stability maps side by side
12 Gallery Displays all identified place cells: trajectory with events (top row) and smoothed rate map with place-field contour (bottom row)
13 Interactive unit browser Browse every unit with full detail panels; toggle between all units and place cells only; requires Jupyter Lab
14 Save bundle Writes a self-contained .camap bundle with all results

Place-cell analysis: the four questions

A cell passes as a place cell only if it passes both the significance test (Questions 1–3) and the stability test (Question 4).

# Question How it is tested What failure means
1 Where does it fire? Activity binned by position divided by occupancy time; smoothed with Gaussian kernel → rate map No spatial structure visible → likely not a place cell
2 Is it more spatial than chance? Spatial information (SI) in bits/spike; higher = more concentrated firing Low SI → activity spread uniformly across arena
3 Could SI arise by chance? 1,000 circular shifts of the activity trace relative to position; compare observed SI to shuffle distribution → p-value Non-significant (p > threshold) → spatial structure could be random timing
4 Is the field stable? Split session into first and second halves; compute separate rate maps; correlate them Low correlation → field is not consistent; likely noise or transient activity
Reading the summary output: After ds.analyze_units(), the table shows total units, significant units (passed Questions 2–3), stable units (passed Question 4), and place cells (passed all four). A typical healthy session in CA1 might show 10–30% of units as place cells. A very high percentage (>50%) may indicate insufficient spatial coverage of the arena; a very low percentage (<5%) may indicate poor signal quality.

Configuration files

Two YAML files in capstone/config/ control the analysis parameters. You should not need to edit these for the prerecorded session, but you will need to update data_paths.yaml if using your own session data.

File What it controls
arena_config.yaml Neural sampling rate, speed threshold for movement filtering, spatial bin count, minimum occupancy, smoothing kernel size, shuffle count, spatial information threshold, place-field peak percentage
data_paths.yaml Paths to all input files; arena calibration (pixel bounds, arena size in mm, camera height); bodypart name for position tracking

Every Time You Return

Steps 1–3 of the installation are permanent. Each time you open a new Terminal window to continue working, run: 

cd /Users/$(whoami)/Downloads/workshop-processing-and-analysis-main
source .venv/bin/activate
jupyter lab

Troubleshooting

The Workshop kernel is not available in Jupyter Lab
Make sure your virtual environment is active (you see (.venv) in your prompt). Then re-run: python -m ipykernel install --user --name workshop --display-name "Workshop" and restart Jupyter Lab.
A notebook cell produces ModuleNotFoundError
The virtual environment is not active or you are using the wrong kernel. In the notebook, go to Kernel → Change Kernel and select Workshop. If Workshop is not listed, see above.
run_cadecon.py or run_catune.py says it cannot find C.zarr
Stage 1 (Minian) has not been run yet, or the output is in a different session folder. Either complete Stage 1 first or run python scripts/get_data.py --what minian_out to download just the Minian output from UCLA Dataverse.
The capstone notebook errors on match_events()
The timestamp files are missing or have not been converted yet. Check that raw/neural_timestamp.csv and raw/behavior_timestamp.csv exist in your session folder (run python scripts/get_data.py --what raw to download them), and that the timestamp conversion cell earlier in the capstone ran successfully.
No place cells are found
First check the timestamp diagnostic plot — a non-linear or gapped timestamp ramp indicates a recording problem that will corrupt the sync. Then check the occupancy map — if large parts of the arena are marked invalid, behavioral coverage may be insufficient. Finally, check the rate maps manually — if activity is present but diffuse, the animal may not have a place-cell representation in this region, or expression quality may be low.
The capstone runs but activity.npy shape mismatch error appears
The number of rows in activity.npy must equal the number of units in Minian's output, and the number of columns must equal the number of neural frames. This means calab was run on a different session or trace than Minian's current output. Re-run Stage 2 to regenerate activity.npy from the correct C.zarr.
FileNotFoundError when opening the Minian notebook
The data has not been downloaded yet. Run python scripts/get_data.py and wait for it to finish before reopening the notebook.

See Also

Retrieved from "https://miniscope.org/index.php?title=Test_page&oldid=3540"