CAT-SG: A Large Dynamic Scene Graph Dataset

for Fine-Grained Understanding of Cataract

Surgery

Felix Holm

1,2,3[0000−0002−8493−9941]

, Gözde Ünver

1,2[0009−0000−5219−8926]

Ghazal Ghazaei

2[0000−0002−0792−3583]

, and Nassir Navab

1,3

Chair for Computer-Aided Medical Procedures, Technical University Munich,

Germany

Carl Zeiss AG, Munich, Germany

Munich Center for Machine Learning (MCML)

felix.holm@tum.de

Abstract. Understanding the intricate workﬂows of cataract surgery re-

quires modeling complex interactions between surgical tools, anatomical

structures, and procedural techniques. Existing datasets primarily ad-

dress isolated aspects of surgical analysis, such as tool detection or phase

segmentation, but lack comprehensive representations that capture the

semantic relationships between entities over time. This paper introduces

the Cataract Surgery Scene Graph (CAT-SG) dataset, the ﬁrst to pro-

vide structured annotations of tool-tissue interactions, procedural vari-

ations, and temporal dependencies. By incorporating detailed semantic

relations, CAT-SG oﬀers a holistic view of surgical workﬂows, enabling

more accurate recognition of surgical phases and techniques. Addition-

ally, we present a novel scene graph generation model, CatSGG, which

outperforms current methods in generating structured surgical represen-

tations. The CAT-SG dataset is designed to enhance AI-driven surgical

training, real-time decision support, and workﬂow analysis, paving the

way for more intelligent, context-aware systems in clinical practice. The

dataset is available at github.com/felixholm/CAT-SG.

Keywords: Scene Graphs · Surgical Data Science · Dataset · Cataract

Surgery

1 Introduction

Advancements in machine learning and computer vision are transforming com-

puter-assisted surgery, enabling precise planning, real-time decision support, and

automated workﬂow analysis. While progress has been made in surgical video

analysis for tasks like tool detection, phase segmentation, and triplet recognition,

achieving a holistic yet ﬁne-grained understanding of surgical workﬂows remains

a challenge.

Scene graphs oﬀer a robust framework for modeling structured interactions

within complex environments, representing entities as nodes and their relation-

ships as edges. In computer vision research, they are increasingly recognized as

2 F. Holm et al.

valuable tools for providing structured, ﬁne-grained, and human-readable rep-

resentations that yield holistic insights into dynamic scenes [11,14,16]. There-

fore, the application of dynamic scene graph representations holds signiﬁcant

promise for revolutionizing surgical workﬂow analysis by encoding surgical scenes

through surgical elements, their spatio-temporal features, and interactions. Al-

though prior research has explored scene graph-based approaches for surgical

workﬂow recognition [15,10,12], these eﬀorts have been constrained by the limi-

tations of existing datasets lacking ﬁne-grained semantic interactions necessary

for detailed procedural analysis.

Cataract surgery, one of the most common and intricate ophthalmic proce-

dures, exempliﬁes the need for structured surgical representations. The success

of phacoemulsiﬁcation, a widely used cataract removal technique, depends on

precise tool-tissue interactions, such as controlled ultrasonic energy application

for lens fragmentation and delicate intraocular manipulation. Cataract proce-

dures involve ﬁne-grained motion dynamics and subtle anatomical interactions,

which existing datasets fail to systematically capture, hindering the development

of automated workﬂow analysis solutions.

To address this gap, we introduce the Cataract Surgery Scene Graph (CAT-

SG) dataset, a novel resource derived from the CATARACTS dataset [1]. CAT-

SG provides structured representations of surgical elements, encompassing their

geometric, temporal, and semantic interactions, with a particular focus on tool-

tissue relationships. It also introduces a new downstream task of surgical tech-

nique recognition, complementing the existing surgical phase annotations in

CATARACTS.

Our contributions are summarized as follows:

CAT-SG Dataset: We present the ﬁrst Cataract Surgery Scene Graph dataset,

with over 1.8 Million annotated relations. Additionally, CAT-SG provides novel

annotations for surgical techniques, enabling a more comprehensive understand-

ing of procedural variations beyond traditional phase recognition.

Comprehensive Benchmarking and Evaluation: To demonstrate the util-

ity of CAT-SG, we establish benchmarks for the three key tasks of our dataset:

scene graph generation, phase recognition and technique recognition, demon-

strating the dataset’s applicability across various surgical workﬂow tasks.

Novel Scene Graph Generation Model (CatSGG): Additionally, we pro-

pose CatSGG, a new scene graph generation model leveraging large-scale pre-

training and spatio-temporal attention to eﬃciently generate structured rep-

resentations of surgical interactions outperforming the state-of-the-art method

ORacle [23] on CAT-SG.

2 CAT-SG Dataset

2.1 CATARACTS

We introduce a novel dataset named CAT-SG that extends the publicly avail-

able CATARACTS dataset [1] with scene graph annotations. The CATARACTS

CAT-SG 3

Skin

Pupil

Iris

Surgical

Tape

Cornea

Secondary

Knife

Hand

Eye

Retractors

holding

cutting

Close to

Skin

Pupil

Iris

Surgical

Tape

Cornea

Lens

Injector

Eye

Retractors

activation

inserting

Close to

Bonn

Forceps

holding

Skin

Pupil

Iris

Surgical

Tape

Cornea

Phaco

Handpiece

Micro-

manipulator

Eye

Retractors

holding

activation

inserting

Close to

Step: Incision

Step: Nucleus Breaking

Step: Lens Injection

Te ch n iqu e : D ivi d e a nd C onq u er

Fig. 1. Samples from one video of CAT-SG, showing Image and Scene Graph including

semantic and geometric relations

dataset consists of 50 high-resolution videos of cataract surgeries, collected at

Brest University Hospital. Each video captures a complete surgical procedure

with a duration of 11 minutes on average (min: 6 min, max: 40 min). With over

nine hours of annotated surgical footage, CATARACTS provides a robust basis

for developing machine learning models for surgical video understanding [1].

2.2 Annotations

The scene graph annotations were created through a manual labeling process

conducted by 9 trained student annotators. The annotators reviewed the videos

and interactively marked each instance of a surgical tool interacting with an

anatomical structure, followed by an iterative expert review process to verify the

validity of our labels. After over 1200 combined hours of annotating, this process

resulted in a detailed temporal mapping of tool-tissue interactions throughout

the surgeries. The interactions are classiﬁed into 8 categories shown in Table 1.

The following 29 objects were considered for annotations: Pupil, Surgical

Tape, Hand, Eye Retractors, Iris, Skin, Cornea, Hydrodissection Cannula, Vis-

Table 1. Relations annotated in CAT-SG

Relation Example Samples

Holding e.g., forceps ﬁxating sclera 13,380

Activation e.g., phaco-handpiece using emulsiﬁcation 44,552

Pushing e.g., micro manipulator pushing & rotating nucleus 3,874

Pulling e.g., cystotome pulling on capsular bag ﬂap 11,895

Cutting e.g., corneal incisions using knife 1,925

Inserting tool is inserted or retracted through the primary or

secondary incision in the cornea

34,016

Retracting

23,886

Close to spatial proximity of one object to another 1,677,724

4 F. Holm et al.

Table 2. Size of comparable datasets and CAT-SG

Avg. Annotated Unique Unique Annotated

Dataset Videos Duration Frames Objects Relations Relations

4D-OR 10 11 min 6,743 11 14 103,740

CholecT45 45 33 min 90,489 20 9 127,385

CAT-SG 50 11 min 164,162 29 9 1,811,252

coelastic Cannula, Capsulorhexis Cystotome, Rycroft Cannula, Bonn Forceps,

Primary Knife, Phacoemulsiﬁcation Handpiece, Lens Injector, Irrigation/Aspi-

ration Handpiece, Secondary Knife, Micromanipulator, Capsulorhexis Forceps,

Suture Needle, Needle Holder, Charleux Cannula, Vitrectomy Handpiece, Mendez

Ring, Marker, Troutman Forceps, Cotton, Iris Hooks, Vannas Scissors.

We sample and annotate the 50 videos in our dataset at a temporal reso-

lution of 5 fps, making CAT-SG the largest dataset among comparable eﬀorts

for detailed surgical scene understanding (Table 2). We also use pseudo-labelled

segmentation masks based on the CaDIS dataset to provide a grounding to each

object in our annotations (position, size, bounding box). We will also release

these masks in addition to our dataset.

Use Cases: CAT-SG is designed to support detailed surgical video understand-

ing through structured scene graph representations oﬀering the following tasks:

Surgical Scene Graph Generation: This task involves automatically con-

structing structured scene graphs from surgical videos. Given a sequence of

frames, the goal is to detect surgical instruments, anatomical structures, and

their interactions over time. Successful models must not only recognize objects

but also infer their relationships, enabling a deeper understanding of surgical

workﬂow and tool usage patterns.

Surgical Workﬂow Recognition: Our dataset enables surgical workﬂow recog-

nition by leveraging the predeﬁned 19 surgical steps from the CATARACTS

dataset [1]. With the integration of scene graph annotations, this process gains

additional structure and granularity, reinforcing its role as a fundamental task

in surgical video understanding.

Surgical Technique Recognition: To enhance surgical video analysis, we

add new annotations distinguishing two nucleus-breaking techniques: “Stop and

Chop” and “Divide and Conquer.” Each procedure in the dataset is labeled as

using one of these two techniques during the nucleus breaking step. Identifying

these techniques requires a nuanced understanding of tool movements and proce-

dural dynamics, making this a complex and valuable benchmark for automated

surgical workﬂow analysis.

3 CatSGG Scene Graph Generation Method

We introduce a novel scene graph generation method comprising of two steps:

1) entity localization and base graph construction, 2) geometric and semantic

CAT-SG 5

8 frames = 1 chunk

…

Pooled Query

Embeddings

…

Pair

Proposals

Skin

Pupil

Iris

Surgical

Tape

Cornea

Secondary

Knife

Bonn

Forceps

cutting

Close to

holding

geometric

relations

semantic

relations

Fig. 2. Overview of the CatSGG+ Pipeline: an input video chunk is fed into

Mask2Former with a video-based backbone to extract rich spatio-temporal features.

The extracted query embeddings are pooled and used to generate pair proposals, which

are then processed by a relation predictor and classiﬁer to infer semantic relations. Ge-

ometric relations are inferred from the predicted mask.

relation prediction.

Annotation-eﬃcient Entity Localization While bounding boxes provide

coarse localization, they lack the spatial precision needed for modeling surgical

interactions in scene graphs. We leverage segmentation models instead, as they

oﬀer ﬁne-grained instance localization, enabling more accurate spatial reasoning

and relationship extraction. Surgical video segmentation could also be challeng-

ing due to the long duration of procedures, e.g. cataract surgeries typically last

around 11 minutes, making manual annotation and computational processing at

high fps highly burdensome. To address this, we propose an annotation-eﬃcient

solution for instance segmentation of surgical elements within CATARACTS

frames by leveraging state-of-the-art segmentation and video-language pretrain-

ing. Our approach builds upon Mask2Former (M2F) [4] while incorporating

domain-speciﬁc prior knowledge from Watch&Learn [8], a large-scale video-

language pretraining framework. Speciﬁcally, we replace the standard back-

bone of M2F with a ﬁnetuned VideoSwin [13], which was trained within the

VALOR framework [3] alongside a BERT encoder [6] for video-language align-

ment. We opt for this model over the original VideoSwin because its pretraining

on 2,900 YouTube cataract surgery videos [8] provides strong domain priors,

allowing eﬀective segmentation with fewer annotated samples. Moreover, the

spatio-temporal design of VideoSwin enhances the model’s awareness of cataract

surgery dynamics, further improving segmentation performance. By integrating

VideoSwin into M2F, we adapt the segmentation pipeline to better capture the

temporal and contextual dependencies in surgical videos while requiring fewer

annotated frames.

Geometric Relation Prediction: The “Close to” relation, which captures spa-

tial proximity, is determined based on M2F segmentation predictions. Speciﬁ-

cally, we identify instances with adjacent masks (touching boundaries) as being

6 F. Holm et al.

close to each other, following the approach in [10].

Semantic Relation Prediction: The VideoSwin backbone of our M2F pro-

cesses 8 consecutive frames as a chunk, with segmentation and scene graph gen-

eration performed only on the last frame. Consequently, our backbone inherently

captures temporal information across frames. Inspired by query embedding solu-

tions [17,18,20,22,21,19], we leverage the output query vectors (q) from M2F for

relation prediction. Each query vector encodes class-speciﬁc and spatial infor-

mation for a single instance. To model pairwise relationships, we construct pair

proposal embeddings by concatenating the queries of two instances. Since seman-

tic relations occur only between tool-tool or tool-anatomy pairs, we incorporate

dataset priors to construct these pair proposals eﬃciently. Relation prediction

consists of two tasks: (1) detecting whether a relation exists (e ≥ 0.5), and (2)

classifying the relation type. Following [20], we employ: A 2-layer model for bi-

nary relation existence prediction (f

existence

), followed by a 3-layer multi-label

classiﬁcation model (f

classiﬁcation

) for relation type prediction.

pair

= [q

; q

] where i = j, i ∈ N

tool

, j ∈ N

tool

∪ N

anatomy

e = σ(f

existence

(pair

)), e ∈ [0, 1]

if e ≥ 0.5, c = f

classiﬁcation

(pair

)

(1)

We use binary cross-entropy loss with sigmoid activations for both models.

Both models operate on the pair embeddings derived from the query vectors

of the last frame in the chunk. We refer to this method as CatSGG. To further

enhance temporal consistency, we extend this approach by aggregating query

vectors from all frames in the chunk. Inspired from [20], we apply max pooling

on same-class queries across these frames to obtain a single representative vector

per class.

These temporally enriched vectors are then used to create pair embeddings for

relation predictions. We refer to this temporally aware extension as CatSGG+.

4 Experiments

CatSGG: Semantic Segmentation: From preliminary experiments we ﬁnd

that training M2F with the pretrained VideoSwin backbone from[8] achieves

equal performance with much faster convergence and improved generalizability

by subsampling the training data to 18 chunks per training video. Using this

conﬁguration we train M2F on the 29 classes from CAT-SG, by merging pseudo-

masks based on CaDIS [9] with CAT-SG annotations. The model achieves a

mIoU of 92.12%, demonstrating its ability to generate initial scene graph nodes.

CatSGG: Semantic Relation Prediction: Using the frozen pretrained seg-

mentation model, we train the relationship prediction models with 18 chunks

per training video. Since semantic relations are relatively infrequent, we apply

sampling constraints to ensure that each selected chunk contains at least one

semantic relation. We apply a 0.5 threshold to ﬁlter predictions from both the

CAT-SG 7

relation existence and classiﬁcation models.

ORacle [23]: We use ORacle as a benchmark, representing the most recent

SOTA method in scene graph generation within the surgical setting. ORacle

is a large vision-language model and does not require bounding boxes or scene

graph grounding, which makes it highly versatile and easily adaptable to various

domains. ORacle is originally developed for the 4D-OR dataset, consisting of

external views captured by overhead cameras in the operating room (OR). We

train and evaluate ORacle’s single view variant, both with and without tem-

porality (denoted as ORacleSV and ORacleSVT, respectively). To account for

the distribution of relation classes, we report F1 scores for each class, as well as

micro- and macro-averaged F1.

Downstream Task Baselines: Since semantic scene graphs connect the graph

and language domain, they can be used both in Graph Neural Networks (GNNs)

and Large Language Models (LLMs). We present baselines for both approaches,

leveraging our scene graphs to recognize surgical workﬂows and techniques. We

report Accuracy and F1 score as the well-established metrics for these tasks [7].

GNN: We use a 3-layer GATv2 [2] model as our GNN baseline. Like [10], we

encode node groundings (i.e., position and size in the image) as part of the

node features. Temporality is captured by connecting scene graphs from diﬀer-

ent timesteps with temporal edges between nodes of the same class, creating a

dynamic scene graph.

LLM: We ﬁne-tune a Llama 3.2 3B model for our downstream tasks using QLora

and its recommended parameters [5]. To represent our scene graphs, we prompt

the model with a list of entities and relations, along with their groundings pro-

vided as text (e.g., “<Object> at (<x>, <y>) with size <size>”). Temporality

is incorporated through a history mechanism: the prompts include scene graphs

from the last N steps prior to the current scene graph that needs to be classiﬁed.

5 Results & Discussion

Scene Graph Generation. Table 3 presents the results for scene graph gen-

eration on CAT-SG. Our CatSGG outperforms the SOTA ORacle by over 8

percentage points (p.p.) on Macro-F1. The inclusion of longer-term temporal

context in CatSGG+ further improves performance, allowing the model to bet-

ter capture the dynamically evolving semantic relations over time. Surprisingly,

adding temporality to ORacle (ORacleSVT) leads to a performance drop. We

hypothesize this is due to ORacle aggregating its predictions over time, which

can propagate mispredictions, thereby degrading overall performance. A video

of qualitative results for CatSGG+ is attached in the supplementary.

Surgical Workﬂow Recognition. As shown in Table 4, introducing se-

mantic relations leads to improvements in both accuracy and F1, surpassing

the baseline from [10] by over 5 p.p. This demonstrates that explicitly encoding

8 F. Holm et al.

Table 3. Scene Graph Generation Results on CAT-SG.

F1 Score for each Relation Overall

Close to

Holding

Activation

Pushing

Pulling

Cutting

Inserting

Retracting

"none"

Micro F1

Macro F1

ORacleSV 67.03 3.74 41.76 1.57 33.08 31.06 39.75 15.00 78.83 72.45 34.65

ORacleSVT 57.64 19.82 35.93 15.58 32.99 14.74 42.16 17.10 74.48 66.19 34.49

CatSGG 91.63 7.52 45.81 4.32 43.09 49.23 37.74 7.26 92.10 89.78 42.08

CatSGG+ 91.63 8.60 44.90 0.00 42.86 46.04 39.32 22.55 92.08 89.78 43.11

tool–anatomy relationships is beneﬁcial for recognizing workﬂow stages that of-

ten hinge upon the correct sequence of instrument interactions. Notably, LLMs

performed competitively in our experiments without temporality, but were not

able to leverage temporal context or groundings for improved performance, de-

spite being much larger models.

Table 4. Surgical Workﬂow Recognition Results

Temporal Semantic Spatial

Window Relations Encoding Accuracy F1

Holm et al. [10] 1 frame ✗ CATARACTS ✓ 65.56 52.24

GATv2 1 frame ✓ CAT-SG ✓ 70.81 56.02

Llama 3.2 3B 1 frame ✓ CAT-SG ✗ 67.70 53.98

Llama 3.2 3B 1 frame ✓ CAT-SG ✓ 69.13 53.87

Holm et al. [10] 30 frames (90 s) ✗ CATARACTS ✓ 73.77 64.93

GATv2 30 frames (90 s) ✓ CAT-SG ✓ 78.63 70.15

Llama 3.2 3B 30 frames (90 s) ✓ CAT-SG ✗ 29.02 5.52

Surgical Technique Recognition.Table 5 shows that our GATv2 model,

utilizing temporal graphs sampled at 5 fps, achieves the highest accuracy and

F1 scores. Interestingly, longer temporal windows sampled with lower resolution

(50 s at 1 fps) result in lower accuracy, suggesting that the ﬁne-grained instru-

ment motions crucial for distinguishing speciﬁc techniques occur within shorter

time spans. This highlights the value of our dataset being annotated at 5 fps.

Furthermore, ablating spatial features degrades performance, conﬁrming that

precise tool positioning is vital for diﬀerentiating lens-breaking strategies.

Table 5. Surgical Technique Recognition Results

Temporal Window Spatial

(sampled fps) Features Accuracy F1

GATv2 10 s (5 fps) ✓ 68.75 ± 4.11 48.40 ± 1.72

GATv2 50 s (1 fps) ✓ 66.34 ± 1.38 41.31 ± 1.83

GATv2 10 s (5 fps) ✗ 64.48 ± 1.15 44.52 ± 1.58

CAT-SG 9

6 Conclusion

In this paper, we introduce CAT-SG, a large, ﬁne-grained scene graph dataset

speciﬁcally designed to understand complex workﬂows in cataract surgery. By

capturing not only the presence of surgical instruments and anatomical struc-

tures but also their intricate interactions, CAT-SG provides an unprecedented

level of detail for modeling real-world procedural variations. Alongside the dataset,

we propose CatSGG, a novel scene graph generation model that leverages pre-

trained video backbones and eﬃciently processes both spatial and temporal in-

formation. The holistic view oﬀered by CAT-SG has the potential to enhance a

wide range of applications, from automated skill assessment and decision sup-

port to advanced surgical training. Furthermore, this structured representation

of surgical procedures promotes more transparent and explainable approaches,

as each relationship can be precisely interpreted in a clinical context. Ultimately,

this can improve patient outcomes and support surgeons in delivering safer, more

eﬃcient care.

Acknowledgments. The authors would like to thank Carl Zeiss AG for support of

this research.

Disclosure of Interests. The authors have no competing interests to declare that

are relevant to the content of this article.

References

1. Al Hajj, H., Lamard, M., Conze, P.H., Roychowdhury, S., Hu, X., Maršalkait˙e,

G., Zisimopoulos, O., Dedmari, M.A., Zhao, F., Prellberg, J., Sahu, M., Galdran,

A., Araújo, T., Vo, D.M., Panda, C., Dahiya, N., Kondo, S., Bian, Z., Vahdat,

A., Bialopetravičius, J., Flouty, E., Qiu, C., Dill, S., Mukhopadhyay, A., Costa,

P., Aresta, G., Ramamurthy, S., Lee, S.W., Campilho, A., Zachow, S., Xia, S.,

Conjeti, S., Stoyanov, D., Armaitis, J., Heng, P.A., Macready, W.G., Cochener,

B., Quellec, G.: Cataracts: Challenge on automatic tool annotation for cataract

surgery. Medical Image Analysis 52, 24–41 (2019). https://doi.org/https://doi.

org/10.1016/j.media.2018.11.008, https://www.sciencedirect.com/science/article/

pii/S136184151830865X

2. Brody, S., Alon, U., Yahav, E.: How attentive are graph attention networks? In:

International Conference on Learning Representations (2022), https://openreview.

net/forum?id=F72ximsx7C1

3. Chen, S., He, X., Guo, L., Zhu, X., Wang, W., Tang, J., Liu, J.: Valor: Vision-

audio-language omni-perception pretraining model and dataset. arXiv preprint

arXiv:2304.08345 (2023)

4. Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention

mask transformer for universal image segmentation. In: Proceedings of the

IEEE/CVF conference on computer vision and pattern recognition. pp. 1290–1299

(2022)

5. Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Eﬃcient ﬁnetun-

ing of quantized llms. arXiv preprint arXiv:2305.14314 (2023)

10 F. Holm et al.

6. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidi-

rectional transformers for language understanding. In: Proceedings of the 2019

conference of the North American chapter of the association for computational

linguistics: human language technologies, volume 1 (long and short papers). pp.

4171–4186 (2019)

7. Funke, I., Rivoir, D., Speidel, S.: Metrics matter in surgical phase recognition

(2023), https://arxiv.org/abs/2305.13961

8. Gastager, D., Ghazaei, G., Patsch, C.: Watch and learn: Leveraging expert knowl-

edge and language for surgical video understanding. ArXiv abs/2503.11392

(2025), https://api.semanticscholar.org/CorpusID:277044026

9. Grammatikopoulou, M., Flouty, E., Kadkhodamohammadi, A., Quellec, G., Chow,

A., Nehme, J., Luengo, I., Stoyanov, D.: Cadis: Cataract dataset for image seg-

mentation. arXiv preprint arXiv:1906.11586 (2019)

10. Holm, F., Ghazaei, G., Czempiel, T., Özsoy, E., Saur, S., Navab, N.: Dynamic

scene graph representation for surgical video. In: Proceedings of the IEEE/CVF

international conference on computer vision. pp. 81–87 (2023)

11. Ji, J., Krishna, R., Fei-Fei, L., Niebles, J.C.: Action genome: Actions as composi-

tions of spatio-temporal scene graphs. In: Proceedings of the IEEE/CVF conference

on computer vision and pattern recognition. pp. 10236–10247 (2020)

12. Köksal, Ç., Ghazaei, G., Holm, F., Farshad, A., Navab, N.: Sangria: Surgical

video scene graph optimization for surgical workﬂow prediction. arXiv preprint

arXiv:2407.20214 (2024)

13. Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., Hu, H.: Video swin trans-

former. In: Proceedings of the IEEE/CVF conference on computer vision and pat-

tern recognition. pp. 3202–3211 (2022)

14. Luo, Z., Durante, Z., Li, L., Xie, W., Liu, R., Jin, E., Huang, Z., Li, L.Y., Wu,

J., Niebles, J.C., Adeli, E., Fei-Fei, L.: MOMA-LRG: Language-reﬁned graphs for

multi-object multi-actor activity parsing. In: Thirty-sixth Conference on Neural

Information Processing Systems Datasets and Benchmarks Track (2022), https:

//openreview.net/forum?id=eJhc_CPXQIT

15. Murali, A., Alapatt, D., Mascagni, P., Vardazaryan, A., Garcia, A., Okamoto, N.,

Mutter, D., Padoy, N.: Encoding surgical videos as latent spatiotemporal graphs

for object and anatomy-driven reasoning. In: International Conference on Medi-

cal Image Computing and Computer-Assisted Intervention. pp. 647–657. Springer

(2023)

16. Rodin, I., Furnari, A., Min, K., Tripathi, S., Farinella, G.M.: Action scene graphs

for long-form understanding of egocentric videos. In: Proceedings of the IEEE/CVF

Conference on Computer Vision and Pattern Recognition. pp. 18622–18632 (2024)

17. Wang, G., Li, Z., Chen, Q., Liu, Y.: Oed: towards one-stage end-to-end dynamic

scene graph generation. In: Proceedings of the IEEE/CVF Conference on Com-

puter Vision and Pattern Recognition. pp. 27938–27947 (2024)

18. Wang, J., Wen, Z., Li, X., Guo, Z., Yang, J., Liu, Z.: Pair then relation: Pair-net

for panoptic scene graph generation. IEEE Transactions on Pattern Analysis and

Machine Intelligence (2024)

19. Yang, J., Ang, Y.Z., Guo, Z., Zhou, K., Zhang, W., Liu, Z.: Panoptic scene graph

generation. In: European Conference on Computer Vision. pp. 178–196. Springer

(2022)

20. Yang, J., Peng, W., Li, X., Guo, Z., Chen, L., Li, B., Ma, Z., Zhou, K., Zhang, W.,

Loy, C.C., et al.: Panoptic video scene graph generation. In: Proceedings of the

IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18675–

18685 (2023)

CAT-SG 11

21. Zhou, Z., Shi, M., Caesar, H.: Hilo: Exploiting high low frequency relations for

unbiased panoptic scene graph generation. In: Proceedings of the IEEE/CVF In-

ternational Conference on Computer Vision. pp. 21637–21648 (2023)

22. Zhu, F., Yang, J., Jiang, H.: Towards ﬂexible visual relationship segmentation.

Advances in Neural Information Processing Systems 37, 107633–107661 (2025)

23. Özsoy, E., Pellegrini, C., Keicher, M., Navab, N.: Oracle: Large vision-language

models for knowledge-guided holistic or domain modeling. In: International Con-

ference on Medical Image Computing and Computer-Assisted Intervention. pp.

455–465. Springer (2024)