Quantifying the human visual exposome with vision language models

Ethics approval was obtained from the Ethics Committee of the University of Graz (reference number: GZ. 39/168/63 ex 2024/25). The study was conducted in accordance with the ethical principles of the Declaration of Helsinki and institutional ethical standards. Participants were recruited via social media and personal contact. All participants provided informed consent before study enrollment and received course credits for participation.

Ecological momentary assessment protocol

Of the 115 adults who participated, nine participants provided data from only one alarm. The remaining 106 adults (28 men), who completed the 7-day ecological momentary assessment via the ESMira smartphone application21, were included in all analyses. The app delivered seven random alarms per day between 09:00 a.m. and 08:00 p.m., with a minimum interval of 30 min between prompts. Participants could postpone alarms if necessary. At each alarm, they reported their momentary affective well-being and the perceived greenness of their surroundings. In addition, they were instructed to capture a photograph of their current environment from their own perspective. Participants could choose to skip taking a photograph if they preferred. In total, 2674 images with corresponding momentary self-reports were collected (i.e. M = 25.2 images per participant, SD = 9.96). Assessments began on a random weekday and continued for 7 consecutive days. At the start of the ecological momentary assessment period, participants completed baseline questionnaires covering demographic and health-relevant information, including age, sex and chronic stress.

Mental health-related variables

We assessed momentary affect via 10 items, which were subdivided into positive affect (5 items: active, awake, stimulated, determined, attentive) and negative affect (5 items: angry, hostile, ashamed, nervous, anxious)27, each rated on a five-point Likert scale from 1 (not at all) to 5 (very much). Reliability analyses suggested good within-person and good between-person reliability for positive affect (RCn = 0.64, RkRn = 0.97) and negative affect (RCn = 0.60, RkRn = 0.97)28. Between-person variables (trait Level 2) were computed as each participant’s mean across all momentary assessments; within-person variables (state Level 1) were the affect ratings at each alarm. The mean positive affect over all alarms was M = 2.89 (SD = 0.84), and the mean negative affect was M = 1.31 (SD = 0.49).

Additionally, we assessed momentary perceived greenness with a single item (‘How green do you perceive your current environment?’) rated from not at all green to extremely green on a six-point Likert scale (M = 2.49, SD = 1.51).

In addition to these momentary assessments, we collected chronic stress using the Perceived Stress Scale (PSS)29. The PSS measures the subjective experience of stress during the past weeks30. We used the validated German version31 with ten items rated on a five-point Likert scale from 1 (never) to 5 (very often). Internal consistency was good (Cronbach’s α = 0.88). The mean score was M = 2.90 (SD = 0.72).

Vision-language model analyses for greenness and related visual features

We primarily used the open-weight LLaMA 4 VLM (model: Maverick-17B-128E-Instruct-FP8; ≈17 billion active parameters; temperature = 0.6), selected as a state-of-the-art open-weight model that can be deployed locally without commercial API access, and accessed via a chat completion endpoint. Vision-language models were used as instruction-tuned models in a non-thinking (non-reasoning) configuration and were prompted to return each rating directly in the required structured format, without requesting chain-of-thought traces. The ‘Instruct’ designation of the Qwen3 VL model denotes the non-reasoning variant, as opposed to the separate ‘Thinking’ variant. No chain-of-thought prompting (e.g. ‘think step by step’) was applied, and no chain-of-thought traces were retained or parsed.

The model was prompted to generate quantitative ratings of greenness and visual features indicative of greenness in real-world environments, namely greenness, nature score, plant presence and natural light exposure, as well as a rating for inside/outside for each participant-generated photograph. To prevent potential cross-influences between categories, prompts were submitted individually, with five independent runs performed for each category. Mean scores across the five runs were used for subsequent analyses. Model outputs were parsed and stored in a structured format for subsequent statistical analysis. We conducted reliability analyses similar to those performed on affect, where runs served as items. We found high within-person (RCn = 0.97 –0.99) as well as between-person (RkRn = 0.76–0.94) reliability across all runs. This argues for the calculation of mean scores for subsequent analyses. Detailed results for all five runs, including means and standard deviations, are provided in Supplementary Table 1. Each category was rated using a separate, single-category prompt of identical structure, in which only the category name was substituted. For the greenness-related features, the prompt was (shown here for ‘Greenness’):

For the inside/outside classification, the rating instruction was replaced by ‘use 1 for Inside and 2 for Outside’. Prompt design followed an iterative refinement process. An initial conversational, instruction-heavy prompt style proved inefficient: it increased prompt length and token cost, lengthened runtime, and reduced the model’s focus on task-relevant content.

We therefore adopted a deliberately concise, imperative prompting strategy based on established prompting practices, structured around five principles: (1) a brief statement of the task context, (2) a clear definition of the core rating task, (3) specification of a fixed output format, (4) an explicit instruction not to return additional content, and (5) concise, imperative phrasing. This design yielded consistent ratings in the required, machine-parseable format and was further refined over several minor iterations without subsequent substantive changes. The same prompting strategy was applied to the prompts used for the broader set of literature-derived features (see below).

To assess the generalizability of the approach beyond a single model architecture, the same procedure, including separate prompt submission, five independent runs per category, and averaging of resulting scores, we applied Qwen3 VL (model: Qwen3-VL-235B-A22B-Instruct-FP8; ≈22 billion active parameters; temperature = 0.7). We found similar high within-person (RCn =  0.99) as well as between-person reliability (RkRn = 0.83–0.94) scores across all runs as for the LLaMA4 VLM ratings (see Supplementary Table 1). All models were executed in November 2025 on a locally hosted high-performance computing node equipped with eight Nvidia H200 GPUs and managed through the vLLM inference engine. Within the inference pipeline, each photograph was downscaled on the fly to a maximum long-edge resolution of 1280 pixels (preserving the original aspect ratio) using the Pillow library (version 12.1.0) in Python (version 3.13.2) with Lanczos resampling; the original full-resolution images (2674 photographs, ~37.71 GB) were retained on standard local storage, while the downscaled versions were generated only transiently for inference and not stored. The greenness analyses processed approximately 139.4 million input and 2.8 million output tokens with LLaMA 4, and 91.8 million input and 4.8 million output tokens with Qwen3 VL; the analysis of all 997 literature-derived features with LLaMA 4 processed approximately 5.57 billion input and 112.8 million output tokens.

As an indication of runtime on our hardware (without maximising concurrency, to preserve capacity for other workloads), cumulative inference time was approximately 1.6 h (LLaMA 4) and 2.5 h (Qwen3 VL) for the five-run greenness analyses, and approximately 67 h for the single-run analysis of all 997 literature-derived features.

Vision-language model analyses beyond greenness

We next examined the generalizability of the framework beyond greenness, focusing on an extensive set of contextual features that are empirically linked with mental health-related variables (positive affect, negative affect and stress) and are potentially identifiable by VLMs in participant-generated photographs. To systematically map this conceptual space, we conducted an AI-driven, large-scale exploratory experiment that mined the entire Europe PMC database (~7 million publications) via the EPMC API using the open-weight GPT-OSS-120B model (120 billion parameters; see Supplementary Fig. 3; all codes, prompts, and intermediate data files generated during the literature search and subsequent analyses are publicly available). GPT-OSS-120B is a reasoning-capable model with a configurable reasoning-effort parameter. Feature extraction was run at the default reasoning-effort setting, while factor categorisation and deduplication were run with reasoning_effort set to ‘medium’. Only the model’s final message content was retained for downstream processing. Intermediate chain-of-thought was not stored or used in any downstream analysis.

In the first step, we conducted a keyword-based search to retrieve all publications reporting associations between the mental health-related variables of interest and contextual features potentially identifiable by VLMs. We used Psychology as a mandatory keyword, complemented by optional terms capturing mental health-related variables (i.e. positive affect, negative affect, stress) and contextual features (e.g. environmental, social, biological or living elements). This search yielded 64,597 publications. In the second step, the open-weight GPT-OSS-120B model (temperature = 0.1) automatically extracted and structured relevant information from these publications, yielding 43,263 contextual features reported across 17,380 publications. In the third step, the same LLM (temperature = 0) condensed these multiword contextual features into concise one- to two-word categories suitable for VLM-based assessment.

In the fourth step, after removing non-responses, seven datasets were generated, capturing all combinations of increases and decreases in positive and negative affect and stress, including one dataset containing contextual features for which studies had examined one or more mental health-related variables but reported no significant associations. In the fifth step, the LLM (temperature = 0) was used to cluster conceptually similar categories within each of these seven datasets into broader groups, resulting in 6559 contextual features. After excluding those contextual features without significant associations, which would not allow meaningful replication within our dataset, 6384 contextual features remained. In the sixth step, three separate datasets were created, one each for positive affect, negative affect and stress, combining all findings on increases and decreases and removing duplicates, retaining only those contextual features for which at least three independent studies reported associations with increases or decreases in the respective mental health-related variables, yielding 1123 feature-outcome associations. After merging across the three mental health-related variables and removing duplicates, a final set of 997 unique contextual features remained, which served as the basis for the statistical analyses (for a full list of detected features for each health-relevant variable and expected effect direction, see GitHub repository).

The subsequent VLM-based ratings of the identified 997 contextual features followed the same general procedure as the greenness analyses using the open-weight LLaMA 4 VLM (model: Maverick-17B-128E-Instruct-FP8; ≈17 billion active parameters; temperature = 0.6), with the exception that only a single inference run per category was conducted for computational efficiency. The 997 literature-derived features were rated using an analogous single-category prompt, following the same prompting strategy described above, but with a JSON output format:

Statistical analyses

All statistical analyses, including descriptive, correlational and multilevel modelling procedures, were conducted in R (version 4.4.1)32.

Using multilevel linear models implemented in the lme4 package (version 1.1-35.5)33, we examined the primary research questions of whether VLM-derived greenness indicators (greenness, nature score, plant presence, natural light exposure and inside/outside) (i) corresponded to participants’ self-reported greenness and whether these VLM-based measures were (ii) associated with mental health-relevant factors (i.e. affect). The VLM-derived greenness indicators (aggregated over five runs) served as dependent variables, and participants were included as random effects. For analysis (i), self-reported greenness was entered as the fixed predictor at Level 1 (within-person) and Level 2 (between-person). For analysis (ii), positive and negative affect were entered as fixed predictors at Level 1 (within-person) and Level 2 (between-person). In these analyses, within-person variables (state, Level 1) were person-mean centred to capture fluctuations around each individual’s average level, while between-person variables (trait, Level 2) were computed as each participant’s mean across all momentary assessments. In a complementary analysis, we examined the relationship between aggregated VLM-derived greenness indicators and self-reported chronic stress, an additional mental health-related factor, by calculating Pearson product-moment correlations.

To assess the generalizability of the approach beyond a single model architecture, we applied the identical analytic pipeline using the Qwen3 VL (model: Qwen3-VL-235B-A22B-Instruct-FP8 model; ≈22 billion parameters; temperature = 0.7).

To assess the scalability of the framework, we extended the analyses beyond greenness. Specifically, we examined associations between VLM-derived ratings across a broad range of contextual features, identified through the AI-based literature mining system and mental health-related psychological variables, using the same statistical approach as described above. To illustrate scalability, we computed these analyses for each single feature out of the 1123 identified effects (described by 997 unique categories). We calculated separate analyses for positive affect (133 positive effects, 28 negative effects), negative affect (425 positive effects, 257 negative effects) and perceived stress (178 positive effects, 102 negative effects). A percentage rate of significant findings higher than 5% would argue for a detection rate of expected effects which is higher than chance.