Yafei's Archive

Introducing CRESSim-Neo: A GPU-Accelerated Simulation Engine for Surgical Robotics

Tue, 01 Sep 2026 00:00:00 GMT

import ImageWrapper from '../../../../components/misc/MarkdownImageWrapper.astro'; import ResponsiveImageGrid from '../../../../components/misc/ResponsiveImageGrid.astro'; import YouTubeVideo from '../../../../components/misc/YouTubeVideo.astro'; import framework from './images/framework.png'; import physicsDemo from './images/physics-demo.png'; import sensorOutputs from './images/sensor-outputs.png'; import bloodSuction from './images/blood-suction.png'; import cdcr from './images/cdcr.png'; import thermalCoagulation from './images/thermal-coagulation.png';

High-performance simulation has become a key part of modern robotics. Simulators are not only used for testing control algorithms, but are also increasingly used to generate training data, collect experience for robot learning, and evaluate policies at scale.

Surgical robotics makes this much harder: A surgical scene may contain rigid instruments, deformable soft tissue, sutures or other strand-like structures, and fluids, all interacting with one another. A useful simulator may also need to support operations that are specific to surgery, such as cutting, cauterization, irrigation and suction.

This is a very different setting from many common robotics benchmarks, where a rigid-body dynamics solver is often enough to capture most of the relevant physics.

Existing research has developed sophisticated methods for modeling tissue deformation, suturing, cutting, and other medical interactions. But many of these systems were designed for surgical training (single-scene) rather than large-scale robotics applications.

At the other end of the spectrum, modern GPU-accelerated robotics simulators (e.g., Isaac Sim, SAPIEN, MuJoCo Playground, Genesis World) have made it possible to run large numbers of environments in parallel and connect simulation directly to learning pipelines. These systems work extremely well for locomotion and rigid-body manipulation, but surgical scenes place very different demands on the physics engine.

This leaves an interesting gap: we want the physical flexibility of a surgical simulator, but the scalability and workflow of a modern robotics simulator. CRESSim-Neo is our attempt to bridge these two worlds.

What CRESSim-Neo is

CRESSim-Neo is a GPU-accelerated simulation engine designed specifically for surgical robotics, with a unified physics framework for rigid bodies, deformable soft bodies, fluids, and strand-like structures.

At the core of the simulator is a position-based dynamics (PBD) framework. PBD is particularly attractive for surgical simulation because it provides a relatively unified way to model very different kinds of materials and interactions within the same solver.

On top of this physics layer, CRESSim-Neo provides a batched rasterization pipeline and GPU-resident simulation workflow, so multiple environments can be simulated efficiently and their outputs can be used directly on the GPU without unnecessary host-device copies, such as for synthetic data generation and robot learning.

The goal is not simply to reproduce an existing surgical training simulator on the GPU. We want CRESSim-Neo to serve as a general simulation engine on which new surgical robotics environments, sensing modalities, and learning tasks can be built.

Why build another simulator for surgical robotics?

A natural question is: why build another simulator?

Modern platforms such as Isaac Sim, Warp, and Newton are increasingly flexible, and in principle many surgery-specific features could be implemented on top of them. But our view is that extensibility alone is not enough.

Surgical simulation often requires unusual operations and tight coupling between different parts of the system. We may want to change how constraints are solved, how different materials interact, how simulation data is laid out on the GPU, or how a custom surgical operation is inserted into the timestep. In these cases, working within the assumptions of a large general-purpose framework can become a significant engineering burden.

With CRESSim-Neo, we wanted something simpler: a stack that we can understand and control from top to bottom.

That means keeping the core architecture relatively small, minimizing unnecessary abstraction layers, and making the important parts of the simulation directly accessible. The goal is for researchers to be able to modify the physics, coupling, sensing, rendering, and execution model when the application demands it (through the available interfaces, or, when necessary, by modifying a comparatively small codebase).

This philosophy also shapes some of the lower-level design choices. As we wrote in the paper:

Additional considerations include (1) cross-operating-system and cross-GPU support, (2) modern graphics API usage, (3) cross-API shading language portability, and (4) graphics and compute API interoperability.

These choices are intended to keep the engine portable and to avoid unnecessary dependence on any single graphics or compute stack.

What CRESSim-Neo enables

CRESSim-Neo is designed to support the kinds of interactions and sensing modalities that make surgical robotics different from general manipulation.

Current capabilities include deformable tissue simulation, strand simulation and suturing-style interactions, fluid simulation and suction, cauterization, cable constraints for continuum and cable-driven robots, and simulated ultrasound.

Because these components share the same GPU-resident runtime, they can be combined within the same scene and executed across multiple environments in parallel.

The same pipeline is intended for robot learning and synthetic data generation. Simulation state, rendered observations, rewards, randomization, and other user-defined computations can remain on the GPU, reducing unnecessary data movement between simulation and learning code.

The broader goal is to make it easier to build new surgical robotics tasks without first having to engineer a new simulator around each one.

The engine also produces the sensor data needed by robotics pipelines: RGB images, segmentation masks, depth maps, and synthesized ultrasound. These outputs can be consumed on the GPU alongside the simulation state.

This workflow is demonstrated in a blood-suction task, where the simulated tool, fluid, and tissue evolve together.

We also explore surgery-specific interactions, including cable-driven continuum robots and heat-dependent changes in tissue appearance.

Source code and contributing

CRESSim-Neo is open source, and we are building it as a research platform rather than a fixed collection of tasks or benchmarks.

We are still actively developing the project, and there is a lot we want to improve. But we hope CRESSim-Neo can become a useful foundation for researchers working on robot learning, simulation, and autonomy in surgery.

The code is available here:

::github{repo="yafei-ou/cressim-neo"}

For setup guides and API references, see the CRESSim-Neo documentation.

We would be excited to see what others build with CRESSim-Neo, and to welcome contributions from the community.

Citation

For the full technical details, please cite the CRESSim-Neo paper:

@article{ou2026cressim,
  title={CRESSim-Neo: A Batched GPU Simulation Engine for Surgical Robotics and Robot Learning},
  author={Ou, Yafei and Naheen, Ahnaf and Mussin, Tleukhan and Jarales, Hans and Chacko Moncy, Melwin and Tavakoli, Mahdi},
  journal={arXiv preprint arXiv:2608.25192},
  year={2026}
}

Ghost: 2 Big To Rig - Some Quick Thoughts

Mon, 31 Aug 2026 00:00:00 GMT

import YouTubeVideo from '../../../../components/misc/YouTubeVideo.astro';

I just finished watching Ghost: 2 Big To Rig and really enjoyed it. I actually went to see it twice.

Unfortunately, I couldn't find an IMAX screening in my city 😭

It is impressive to see just how huge the whole production became. The visuals and sound are amazing (as usual).

Having seen Ghost in Vancouver earlier this year, it was especially fun seeing the scale of the show again from a completely different perspective (and noticing some details in the performance that I missed during the actual show), as well as seeing everything going on behind the scenes.

It does feel a little bittersweet knowing Ghost are on hiatus now, but after seeing the scale of everything they have been doing, I can understand why a break is needed.

I will definitely be revisiting this one after the Blu-ray/DVD release, if there is one (I really hope so 🥺).

Watch the official clip for “Umbra” here:

The Cost of Convenience: Ubuntu, Robotics, and the Loss of Control

Sun, 16 Aug 2026 00:00:00 GMT

Over the past several years, I have become increasingly uncomfortable with two related trends: Linux becoming more centralized and opaque in the name of convenience, and robotics researchers becoming increasingly dependent on abstractions they often do not understand underneath. Ubuntu, ROS, Python environments, and containers are all useful technologies. The concern is what happens when useful abstractions become the only layer people know.

This article is an attempt to connect those trends.

Is Ubuntu Becoming Another Windows?

Calling Ubuntu "another Windows" is obviously provocative, and taken literally it is wrong. Ubuntu is still built overwhelmingly from free and open-source software. Its source code can be inspected, modified, redistributed, and forked in ways that Windows cannot. The point of the comparison is not licensing. It is the philosophy of the operating system.

Ubuntu increasingly behaves less like one distribution participating in the Linux ecosystem and more like a vendor-controlled platform.

Traditional Linux distributions tend to preserve an explicit relationship between the user and the system. The package manager matters. The repository matters. The build recipe matters. The dependency graph matters. If I install a Debian package, I generally understand that I am asking the distribution to integrate that software into the system using Debian's packaging model. If I use Arch, I can inspect a PKGBUILD and see, often quite directly, how upstream source turns into the files installed on my machine.

This is not merely aesthetic. It is part of the administrative contract of a Unix-like system: the user should be able to understand what changed, why it changed, and which component is responsible.

Ubuntu has increasingly moved away from that model.

Snap is the clearest example. The objection to Snap is not simply that it is slower, larger, or technically different from .deb packaging. The more important issue is that it changes the relationship between the distribution, the software publisher, and the machine owner. Instead of software primarily being integrated into the distribution, Snap encourages applications to arrive as publisher-prepared bundles distributed through infrastructure closely associated with Canonical. Dependencies are increasingly hidden inside the application artifact rather than expressed as visible parts of the host system.

This changes the question from:

How has this software been integrated into my system?

to:

Does the application run?

That is a very different computing culture.

The distinction becomes even more troubling when Ubuntu blurs the boundary between APT and Snap. A user typing

sudo apt install firefox

has a reasonable expectation that they are choosing the Debian packaging model: APT metadata, .deb packages, distribution-managed dependencies, and software delivered through configured APT repositories.

Yet Ubuntu has used transitional packages and other integration mechanisms to move applications from traditional packages to Snap. From the user's perspective, the familiar package-management interface may remain while the underlying distribution mechanism changes.

One can defend this technically. Canonical can argue that users care about obtaining an application, not about the exact format in which it arrives; maintaining multiple packaging systems is expensive, application publishers want predictable environments, and bundled dependencies can simplify support.

But that defense reveals the philosophical shift.

In a traditional Linux model, the delivery mechanism is part of the administrator's choice. In a platform model, the user chooses the application and the vendor decides how it should be delivered, which is the philosophy of Windows, macOS, Android, and other vendor-managed operating environments.

The same tendency appears elsewhere in Ubuntu. Automatic update mechanisms, Canonical-managed services, Ubuntu Pro, extended security maintenance, entitlement checks, telemetry systems, and vendor infrastructure may each have reasonable individual justifications. Security maintenance costs money. Enterprise customers need predictable support. Automatic updates can protect users who would otherwise run vulnerable software indefinitely.

The concern is not that any one of these features is uniquely malicious. The concern is what they become collectively.

The operating system increasingly establishes an ongoing relationship between the user's machine and the company that produced the distribution. The vendor is no longer merely producing packages and publishing releases. It increasingly participates in application delivery, update policy, support entitlement, lifecycle management, and system services after installation.

For an enterprise customer, this may be exactly what is wanted.

For a traditional Linux user, it raises a more fundamental question:

Why should the company that produced my distribution continue to exercise authority over my machine after I have installed it?

This question becomes especially important because Ubuntu's influence extends far beyond its existing users. Hardware manufacturers support Ubuntu because it has many users. Software vendors test on Ubuntu because customers expect it. Tutorials assume Ubuntu because that is what beginners are likely to run. Research software documents Ubuntu installation because maintaining instructions for ten distributions is expensive.

Every individual decision is rational.

Together, they create a network effect.

More Ubuntu users produce more Ubuntu-specific support. More Ubuntu-specific support makes alternative distributions less convenient. The reduced convenience pushes more users toward Ubuntu. Eventually, software can claim "Linux support" while in practice meaning "we tested this on the currently supported Ubuntu LTS."

Ubuntu therefore does not need a formal monopoly to acquire monopoly-like influence.

Users remain legally free to choose Debian, Arch, Fedora, NixOS, openSUSE, or something else. But freedom in principle may not necessarily be freedom in practice. If hardware documentation, research software, binary releases, vendor support, and third-party tutorials increasingly assume one distribution, the cost of exercising that freedom rises.

ROS is a good example. Windows and macOS may be officially supported, but in practical robotics work the surrounding ecosystem--hardware drivers, vendor SDKs, tutorials, simulation tooling, and third-party ROS packages--is often much more strongly oriented toward Linux. Debian itself is only a Tier 3 platform for ROS 2 Lyrical, while Ubuntu 26.04 is Tier 1. RHEL/Fedora provides another Linux path, but with Tier 2 support it does not occupy anything close to Ubuntu’s position within the robotics community. Thus ROS may be formally cross-platform while Ubuntu remains the de facto platform.

ROS 2 Lyrical lists the following supported platforms even for building from source:

Building from source
--------------------

We support building ROS 2 from source on the following platforms:

* :doc:`Ubuntu Linux 26.04 <Installation/Alternatives/Ubuntu-Development-Setup>`
* :doc:`Ubuntu Linux 26.04 (Pixi) <Installation/Alternatives/Ubuntu-Development-Setup-Pixi>`
* :doc:`Windows 11 <Installation/Alternatives/Windows-Development-Setup>`
* :doc:`RHEL-10/Fedora <Installation/Alternatives/RHEL-Development-Setup>`
* :doc:`macOS <Installation/Alternatives/macOS-Development-Setup>`

Ubuntu can become effectively mandatory without ever being declared mandatory.

The desktop itself illustrates the same tendency. Ubuntu Desktop is not a neutral collection of interchangeable components from which users are expected to construct their preferred environment. It presents a curated product: GNOME, Canonical's modifications, a chosen set of applications, predefined integration, and an intended workflow.

Of course these components can be replaced. Linux remains Linux. Other Ubuntu flavors exist (e.g., Kubuntu), and an experienced user can rebuild large portions of the system.

But there is an important difference between something being technically replaceable and a system being designed around replaceability.

Traditional Linux culture often treats composability as a first-class property. A window manager, display manager, shell, package source, service, or desktop environment is a component. The user chooses it because the machine belongs to the user.

The increasingly product-oriented distribution instead says: this is the experience we have designed; customization occurs around that experience.

Again, this resembles the direction of mainstream proprietary operating systems far more than it resembles the older Unix ideal.

None of this makes Ubuntu equivalent to Windows. The distinction in licensing and source availability remains enormous. Nor does it mean Canonical's engineering decisions are irrational. In many cases Canonical is solving genuine problems created by Linux itself: fragmentation, dependency incompatibility, maintenance burden, security, poor application distribution, and the difficulty of supporting arbitrary combinations of system components.

That is precisely why the trend deserves criticism.

Centralization rarely arrives by announcing itself as centralization. It arrives as a solution to inconvenience.

A centralized application store is easier for publishers. Bundled applications are easier to test. Automatic updates are easier for users. A single supported distribution is easier for hardware vendors. A preselected desktop is easier to document. A commercial support layer is easier for enterprises to purchase.

Every step can make sense locally while producing an unhealthy result globally.

The danger is not that Canonical will suddenly make Ubuntu proprietary. The danger is that Ubuntu becomes so deeply embedded as the default interpretation of "Linux" that the diversity of the wider ecosystem becomes increasingly theoretical, especially in fields such as robotics.

This is therefore a betrayal of a different Linux tradition: that the operating system should remain legible, composable, decentralized, and ultimately subordinate to the person administering the machine.

Ubuntu still gives the user freedom.

The more uncomfortable question is whether it is building an ecosystem in which fewer users understand that freedom, fewer users exercise it, and progressively more infrastructure assumes that they will not.

Roboticists Who Don't Understand Their Computers

Robotics makes the problem especially visible: freedom is of limited practical value to users who do not understand the system they are supposedly free to control. Worse, the increasing abstraction and standardization of the ecosystem can reinforce that lack of understanding.

The easier it becomes to treat Ubuntu, ROS, containers, and package managers as opaque infrastructure, the less incentive there is to understand what happens underneath. In turn, users who lack that understanding become even more dependent on the default platform and its abstractions.

A surprising number of people working in the field do not have a strong grasp of basic computer systems.

This is not a claim that roboticists are unintelligent, nor that everyone working on robotics should be a computer scientist. Robotics naturally attracts people from mechanical engineering, electrical engineering, controls, signal processing, computer vision, and many other disciplines. That diversity is one of the field's strengths.

The problem is that modern robotics has become, to a large extent, a computing discipline without fully accepting the educational consequences of becoming one.

In my own experience, I have encountered researchers and engineers who can work on sophisticated robotics problems but do not understand even the basic mechanics of building software: what compilation does, what linking does, why a linker cannot find a symbol, what a shared library is, or what a build system such as CMake is actually doing. This is not about knowing compiler theory or being able to write a linker. The problem is the absence of a basic mental model of how source code becomes a running program.

The same problem appears in networking, operating systems, permissions, processes, and security. Remote-access services may be exposed casually because getting access to the robot is treated as the objective, while questions about authentication, network boundaries, or who else might be able to reach that service are treated as secondary concerns. Permissions are sometimes bypassed rather than understood. Networking problems are debugged by repeatedly changing settings until something works rather than by reasoning about routes, ports, interfaces, or trust boundaries.

These are not exotic areas of computer science. They are basic properties of the machines on which modern robots depend.

This becomes particularly important because a robot is not simply an algorithm attached to some motors. A modern robot is a networked computer system with actuators attached to it. It may contain multiple computers, cameras, microphones, wireless interfaces, remote-management services, GPUs, embedded controllers, cloud connections, and software from many different vendors. A mistake in an ordinary desktop application may crash the application. A mistake in a robotic system may expose cameras, provide remote control of the machine, or affect something moving in the physical world.

The security problems discovered in Yarbo robotic lawnmowers are an example. Researchers found serious weaknesses involving remote access and privileged credentials. What is striking about such failures is not that they required some obscure cryptographic breakthrough to understand. Yarbo intentionally built remote administrative access, but apparently designed its security model extraordinarily poorly.

It is tempting to describe such incidents simply as cybersecurity failures, but that may be too narrow. They can also reveal a more fundamental lack of computer systems knowledge--or an organization that failed to ensure such knowledge was present in the design and review process. One does not need to be a professional security researcher to understand that the same root password should not be hard-coded across deployed machines, or that remotely accessible control interfaces require careful authentication and isolation.

The organizational problem is equally important. In robotics research groups and startups, an engineer may be hired because of expertise in computer vision, controls, mechanical design, or machine learning and then gradually become responsible for software deployment, networking, remote access, build infrastructure, or system administration simply because somebody has to do it. Expertise in one technical area is quietly treated as if it implies competence in all the others.

It does not.

Knowing how to train a vision model is not the same thing as knowing how to build a computer system. Knowing control theory does not imply understanding operating-system privileges. Expertise in embedded electronics does not automatically imply competence in network security. These are different bodies of knowledge, and robotics increasingly requires them to interact.

ROS both responds to and reinforces this situation.

ROS is enormously valuable precisely because it hides a great deal of systems complexity. It provides communication abstractions, package conventions, launch mechanisms, tooling, and a common structure that allows researchers to focus on the robotics problem rather than repeatedly rebuilding infrastructure.

That is good engineering.

The problem begins when abstraction becomes a substitute for understanding rather than a tool built on top of it. A researcher may learn which ROS command to run without understanding what processes are being created, how packages were built, how libraries are found, how communication occurs between machines, or why changing an environment variable suddenly fixes the system.

As long as the abstraction works, this can appear perfectly sufficient.

Robotics, however, is a field in which abstractions leak constantly. Drivers fail. Devices disappear. Networks become unreliable. Libraries conflict. Timing changes. GPUs run out of memory. Processes crash. Permissions prevent access to hardware. A binary built against one library version is deployed onto another machine. A researcher may then wonder why "it stops working" after simply copy-pasting a built binary onto another machine. Suddenly the researcher has crossed the boundary between "using ROS" and "understanding the computer."

The problem, therefore, is not abstraction itself. Good abstractions are one of the reasons modern robotics is possible. Nor should every robotics researcher be expected to become an operating-system specialist, a network engineer, and a compiler developer.

But there has to be a minimum level of systems literacy below the abstraction.

A person deploying software onto a networked physical machine should understand, at least conceptually, processes, users and privileges, networking, compilation and linking, libraries, dependencies, and basic security boundaries. These are no longer peripheral skills for robotics. They are part of understanding the experimental apparatus itself.

Yet the incentives of robotics research frequently push in the opposite direction. The researcher is rewarded for the new controller, perception algorithm, planning method, or benchmark result. Time spent understanding a linker problem, improving deployment infrastructure, reviewing remote-access security, or making a build system robust rarely produces a paper.

The rational response is therefore to learn only enough computing to make the experiment run.

ROS, Ubuntu, Docker, Python environments, and increasingly sophisticated tooling make this easier than ever. That is useful, but it also means that someone can operate remarkably high in the software stack while having only a vague idea of what is underneath.

The result is a strange contradiction: robotics is one of the most systems-oriented areas of engineering, yet its research culture can allow people to work on extremely complex systems without requiring them to understand the computers at the center of those systems.

When everything works, that knowledge can look unnecessary.

When it does not, the robot quickly reminds us otherwise.

When Everything Becomes `pip install`

This pressure to learn only enough computing to make the experiment run has shaped the tooling culture of modern robotics. Python, Conda, prebuilt wheels, containers, and increasingly complete software environments all reduce the amount of systems knowledge required to get from an idea to a working result.

That is one of their greatest strengths.

It is also where the next problem begins.

Python became dominant in robotics research for good reasons. It shortens the distance between an idea and an experiment. But the workflow around it increasingly encourages users to think of software as something that can simply be installed, imported, and executed, while the native libraries, binary interfaces, compiler toolchains, drivers, and runtime dependencies underneath disappear from view.

A command such as

pip install torch

looks simple, but what it installs is not simply "Python code." Packages such as PyTorch depend heavily on compiled C/C++ and CUDA components. Current PyTorch binaries may bring their own CUDA runtime libraries rather than relying directly on the system CUDA toolkit.

This becomes messy as soon as the Python world interacts with software outside it.

A user may have a Conda environment containing PyTorch and its CUDA runtime while also using C++ code built against the CUDA installation on the host system. Different processes can legitimately use different CUDA runtimes, but problems begin when native extensions, plugins, build environments, or runtime search paths cross those boundaries. Which library gets loaded can then depend on search paths, RPATHs, environment variables, and the order in which libraries are loaded.

At that point, "which CUDA version am I using?" no longer has an obvious answer.

The same problem appears when building Python wheels inside Conda environments. A package may compile a native extension and silently discover headers or shared libraries from the host system. The resulting wheel works on the developer's machine, but it is not really self-contained: it has accidentally acquired dependencies on libraries outside the environment. Later, another version of the same library may appear inside Conda, the environment may be moved to another computer, or the system libraries may change.

The failure then looks mysterious only because the dependency boundary was never understood in the first place.

This is one of the strange properties of the modern Python scientific ecosystem: it gives the appearance of being high-level and self-contained while resting on an increasingly complicated collection of native software underneath. Python package managers solve dependencies at the package level, but native software still ultimately lives in the world of compilers, ABIs, shared objects, dynamic loaders, drivers, and operating-system search paths.

Those worlds do not always align cleanly.

The problem is made worse by the fact that many users do not know they are crossing that boundary at all. A wheel is experienced as a Python package even when most of the important work is being performed by compiled libraries. A Conda environment is experienced as an isolated environment even when software built inside it has linked against the host. A machine-learning library is experienced as a Python library even when its execution depends on a particular relationship between CUDA userspace libraries and the host NVIDIA driver stack.

When this becomes difficult enough, the standard answer is increasingly:

Just use Docker.

Docker is extremely useful, especially for reproducibility and deployment. In robotics it can make sense to distribute different components as separate images, isolate dependencies, pin versions, and deploy the same software across many machines. I have even seen commercial robots delivered with several Docker containers running different parts of the system, and this is becoming a common architectural pattern.

But Docker is also increasingly treated as a universal cure for dependency problems that nobody wants to understand.

A container can make a complicated dependency graph reproducible without making that dependency graph understood.

The GPU case demonstrates this especially well. A Docker image containing CUDA software is not actually independent of the host GPU stack. The NVIDIA kernel driver still lives on the host, and GPU access from containers depends on additional machinery such as the NVIDIA Container Toolkit. The supposedly self-contained container therefore still relies on a carefully maintained boundary between the host driver, device nodes, container runtime, CUDA userspace libraries, and the application inside the container.

The abstraction becomes something like:

host Linux kernel
    ↓
host NVIDIA driver
    ↓
NVIDIA Container Toolkit
    ↓
container runtime
    ↓
CUDA libraries inside the container
    ↓
PyTorch / TensorRT / ROS
    ↓
robotics application

When everything works, the user may only see the last line.

When it stops working, suddenly the difference between the host driver and CUDA runtime matters. So does the container runtime configuration. So do device permissions. So does the way GPU devices are passed into the container.

And robotics makes the container boundary even less clean.

A web service can often live comfortably inside a container and communicate through a network socket. A robot frequently needs direct access to cameras, serial devices, USB devices, CAN interfaces, GPUs, shared memory, real-time scheduling, and sometimes host networking. Each requirement punches another hole through the supposedly isolated environment.

The closer containerized software gets to physical hardware, the less convincing the idea of the completely self-contained container becomes.

Again, this does not mean containers are bad engineering. Quite the opposite: when the boundaries are deliberately understood and designed, they can be one of the best ways to deploy a complicated robotics system.

The problem is using them as a substitute for understanding those boundaries.

There is a recurring pattern:

native dependencies become difficult
        ↓
use a Python environment

Python and native dependencies become difficult
        ↓
use Conda

Conda and system dependencies become difficult
        ↓
use Docker

Docker and hardware integration become difficult
        ↓
add runtime hooks, device mappings, host networking,
privileges, volumes, and vendor-specific tooling

The dependency problem has not disappeared. It has moved.

Meanwhile, the stack has gained another abstraction layer, and the person operating it may understand less of what is happening underneath.

This is why the Pythonic workflow matters beyond language preference. It reflects a broader change in research culture. The dominant goal is increasingly to make software easy to consume: install the package, activate the environment, start the container, run the launch file.

That is enormously productive, and research incentives reward it. The researcher wants to test an algorithm, not spend three days understanding ELF symbol resolution or CUDA library loading.

But there is a cost when this convenience becomes the only model of computing that the researcher knows.

Python itself is not the problem. Conda is not the problem. Docker is not the problem. ROS is not the problem. Each solves a real and difficult problem.

The problem appears when each abstraction is used to cover the previous abstraction without anyone maintaining a clear mental model of the complete system.

At some point, a robotics researcher may be using Python code inside a Conda environment inside a Docker container, calling into C++ and CUDA libraries, communicating through ROS, accessing GPU and hardware devices through runtime-specific mechanisms, on an Ubuntu system whose own software stack they do not fully understand.

Everything may work.

But when it does not, the debugging problem is no longer a Python problem, a ROS problem, a Docker problem, or an Ubuntu problem.

It is a computer-systems problem.

And that is precisely the knowledge the workflow has made easiest to avoid learning.

Conclusion: Convenience, Abstraction, and the Loss of Control

None of the technologies discussed here are inherently the problem. Ubuntu, ROS, Python, Conda, and Docker all became popular because they solve real problems and make difficult work easier.

The problem is what happens when convenience becomes dependence, abstraction becomes opacity, and standardization becomes centralization.

Ubuntu risks turning “Linux” into a platform defined by one vendor. ROS reinforces that platform because doing otherwise is expensive. Robotics culture often rewards getting the experiment to run more than understanding the system that runs it. Pythonic workflows and containers then make it increasingly possible to operate high in the stack without ever learning what lies underneath.

Each step is individually rational. Together, they can produce an ecosystem that is easier to use but harder to understand, easier to standardize but harder to escape, and increasingly dependent on abstractions that fewer people are capable of debugging when they fail. Coding agents may accelerate that trend by making it possible to repair systems through language rather than through understanding.

Perhaps the real question is not whether these tools give us enough freedom.

It is whether we will still know what that freedom means when the abstractions stop working.

Written by Yafei Ou, assisted by ChatGPT (2026-08-15).
© 2026 Yafei Ou. All rights reserved.

便利的代价：Ubuntu、机器人技术与控制权的丧失

Sun, 16 Aug 2026 00:00:00 GMT

过去几年里，我对两个相互关联的趋势越发感到不安：Linux 以便利之名变得越来越集中化和不透明，而机器人研究人员越来越依赖各种抽象，却往往并不理解这些抽象背后的底层原理。Ubuntu、ROS、Python 环境和容器都是有用的技术。但令人担忧的是，当有用的抽象成为人们唯一了解的层面时，会发生什么。

本文试图将这两个趋势联系起来。

Ubuntu 正在变成另一个 Windows 吗？

称 Ubuntu 为“另一个 Windows”显然是一种挑衅，从字面上看也是错误的。Ubuntu 仍然绝大部分由自由和开源软件构建。其源代码可以被检查、修改、再分发和分叉，这是 Windows 所不能的。然而，这里比较的重点不在于开源许可，而在于操作系统的哲学。

Ubuntu 越来越不像一个参与 Linux 生态系统的发行版，而更像一个由厂商控制的平台。

传统的 Linux 发行版倾向于保持用户与系统之间明确的关系。包管理器很重要，软件仓库很重要，构建配方很重要，依赖关系图很重要。如果我安装一个 Debian 包，我大致明白我是在要求发行版使用 Debian 的打包模型将该软件集成到系统中。如果我使用 Arch，我可以查看 PKGBUILD，并且通常能够很直接地看到上游源代码如何变成安装在我机器上的文件。

这不仅仅是审美问题。它是类 Unix 系统管理契约的一部分：用户应该能够理解什么发生了变化、为什么发生变化，以及哪个组件对此负责。

Ubuntu 越来越偏离了这种模式。

Snap 是最明显的例子。对 Snap 的反对并不仅仅在于它更慢、更大，或者技术上与 .deb 打包不同。更重要的问题是，它改变了发行版、软件发布者和机器所有者之间的关系。

软件不再是主要由发行版集成，Snap 鼓励应用程序以发布者准备好的捆绑包形式到达，并通过与 Canonical 密切关联的基础设施进行分发。

依赖项越来越多地隐藏在应用程序工件内部，而不是作为主机系统的可见部分来表达。

这改变了问题的本质，从：

这个软件是如何集成到我的系统中的？

变成了：

这个应用程序能运行吗？

这是一种截然不同的计算文化（computing culture）。

当 Ubuntu 模糊了 APT 和 Snap 之间的界限时，这种区别就更加令人不安了。用户输入

sudo apt install firefox

时，有一个合理的期望：他们选择的是 Debian 打包模型——APT 元数据、.deb 包、由发行版管理的依赖项，以及通过配置好的 APT 仓库交付的软件。

然而，Ubuntu 却使用过渡包和其他集成机制将应用程序从传统包迁移到 Snap。从用户的角度看，熟悉的包管理界面可能保持不变，但底层的分发机制却已经改变。

人们当然可以从技术上为此辩护。Canonical 可以辩称，用户关心的是获取应用程序，而不是它到达的具体格式；维护多个打包系统成本高昂；应用程序发布者想要可预测的环境；捆绑依赖可以简化支持。

但这种辩护恰恰揭示了哲学上的转变。

在传统的 Linux 模型中，交付机制是管理员的选择的一部分。

在平台模型中，用户选择应用程序，而厂商决定它应该如何交付——这正是 Windows、macOS、Android 以及其他由厂商管理的操作环境的哲学。

同样的趋势也出现在 Ubuntu 的其他方面。自动更新机制、Canonical 管理的服务、Ubuntu Pro、扩展安全维护、授权检查、遥测系统和厂商基础设施，每一项可能都有各自合理的理由。安全维护需要成本。企业客户需要可预测的支持。自动更新可以保护那些本来会无限期运行有漏洞软件的用户。

令人担忧的不是其中某一项功能本身怀有恶意，而是它们共同构成了什么。

操作系统越来越多地在用户机器和生产该发行版的公司之间建立一种持续的关系。厂商不再仅仅生产软件包和发布版本。它在安装后越来越多地参与应用程序交付、更新策略、支持授权、生命周期管理和系统服务。

对于企业客户来说，这可能正是他们想要的。

对于传统的 Linux 用户来说，这引发了一个更根本的问题：

为什么生产我的发行版的公司，在我安装之后仍然应该对我的机器行使权力？

这个问题变得尤为重要，因为 Ubuntu 的影响力远远超出了其现有用户。硬件制造商支持 Ubuntu，因为它有很多用户。软件厂商在 Ubuntu 上测试，因为客户期望如此。教程默认使用 Ubuntu，因为那是初学者最可能运行的系统。研究软件只记录 Ubuntu 的安装步骤，因为为十个发行版维护说明文档成本太高。

每一个单独的决策都是理性的。

但它们共同创造了一种网络效应。

更多的 Ubuntu 用户产生了更多 Ubuntu 专属的支持。更多的 Ubuntu 专属支持使得其他发行版变得更不方便。便利性的降低又推动更多用户转向 Ubuntu。最终，软件可以声称“支持 Linux”，而实际上意思是“我们只在当前受支持的 Ubuntu LTS 上测试过”。

因此，Ubuntu 不需要形式上的垄断就能获得类似垄断的影响力。

用户名义上仍然可以自由选择 Debian、Arch、Fedora、NixOS、openSUSE 或其他发行版。但原则上的自由在实践中未必是真正的自由。

如果硬件文档、研究软件、二进制发布、厂商支持和第三方教程越来越多地默认某一个发行版，那么行使这种自由的成本就会上升。

ROS 就是一个很好的例子。Windows 和 macOS 可能被官方支持，但在实际的机器人工作中，周边生态系统——硬件驱动、厂商 SDK、教程、仿真工具和第三方 ROS 包——往往更强烈地面向 Linux。Debian 本身在 ROS 2 Lyrical 中只是第三层（Tier 3）平台，而 Ubuntu 26.04 是第一层（Tier 1）。RHEL/Fedora 提供了另一条 Linux 路径，但作为第二层（Tier 2）支持，它在机器人社区中的地位远不能与 Ubuntu 相提并论。因此，ROS 可能在形式上是跨平台的，而 Ubuntu 仍然是事实上的平台。

甚至对于从源码构建，ROS 2 Lyrical 也只列出了以下受支持的平台：

Building from source
--------------------

We support building ROS 2 from source on the following platforms:

* :doc:`Ubuntu Linux 26.04 <Installation/Alternatives/Ubuntu-Development-Setup>`
* :doc:`Ubuntu Linux 26.04 (Pixi) <Installation/Alternatives/Ubuntu-Development-Setup-Pixi>`
* :doc:`Windows 11 <Installation/Alternatives/Windows-Development-Setup>`
* :doc:`RHEL-10/Fedora <Installation/Alternatives/RHEL-Development-Setup>`
* :doc:`macOS <Installation/Alternatives/macOS-Development-Setup>`

Ubuntu 可以在从未被宣布为强制的情况下，变得事实上不可或缺。

桌面环境本身也体现了同样的趋势。Ubuntu 桌面并不是一个中立的、由可互换组件构成的集合，期望用户从中构建自己偏好的环境。它呈现的是一个精心策划的产品：GNOME、Canonical 的修改、一组选定的应用程序、预定义的集成以及预期的工作流程。

当然，这些组件是可以替换的。Linux 仍然是 Linux。还存在其他 Ubuntu 变体（例如 Kubuntu），有经验的用户可以重建系统的大部分内容。

但某样东西在技术上可替换与一个系统围绕可替换性来设计之间，存在着重要的区别。

传统的 Linux 文化往往将可组合性视为一等属性。窗口管理器、显示管理器、shell、软件源、服务或桌面环境都是一个组件。用户选择它，是因为机器属于用户。

越来越产品化的发行版则说：这是我们设计的体验，定制则需要围绕这个体验进行。

同样，这更接近主流专有操作系统的方向，而远非更古老的 Unix 理想。

这些都不能使 Ubuntu 等同于 Windows。许可证和源代码可用性方面的区别仍然是巨大的。这也不意味着 Canonical 的工程决策是非理性的。在许多情况下，Canonical 正在解决 Linux 本身造成的真正问题：碎片化、依赖不兼容、维护负担、安全性、糟糕的应用程序分发，以及支持任意系统组件组合的困难。

这正是这个趋势值得批评的原因。

集中化很少以宣布自己是集中化的方式到来。它是以解决不便的方案的形式到来的。

集中式应用商店对发布者来说更容易。捆绑应用程序更容易测试。自动更新对用户来说更容易。单一受支持的发行版对硬件厂商来说更容易。预选的桌面环境更容易编写文档。商业支持体系对企业来说也更容易购买。

每一步在局部上都是合理的，但在整体上却产生了一个不健康的结果。

危险不在于 Canonical 会突然将 Ubuntu 变成专有软件。危险在于 Ubuntu 变得如此深入地嵌入为“Linux”的默认解释，以至于更广泛生态系统的多样性变得越来越理论化，尤其是在机器人等领域。

因此，这是对另一种 Linux 传统的背叛：操作系统应该保持可读性、可组合性、去中心化，并且最终服从于管理机器的人。

Ubuntu 仍然给予用户自由。

但令人不安的问题是，它是否正在构建一个生态系统，在其中更少的用户理解这种自由，更少的用户行使这种自由，而越来越多的基础设施默认他们不会行使这种自由。

不懂自己计算机的机器人研究者

机器人技术使这个问题尤为明显：对于不理解他们理应能够自由控制的系统的用户来说，自由的实用价值是有限的。更糟糕的是，生态系统日益增加的抽象化和标准化可能加剧这种理解的缺乏。

将 Ubuntu、ROS、容器和包管理器视为不透明的基础设施越容易，理解底层发生了什么就越没有动力。反过来，缺乏这种理解的用户变得更加依赖默认平台及其抽象。

令人惊讶的是，在这个领域工作的人中，有相当多的人对基本的计算机系统没有扎实的掌握。

这并不是说机器人研究者不聪明，也不是说每个从事机器人技术的人都应该是计算机科学家。机器人技术自然地吸引来自机械工程、电气工程、控制、信号处理、计算机视觉和许多其他学科的人。这种多样性是该领域的优势之一。

问题在于，现代机器人技术在很大程度上已经成为一个计算学科，却没有完全接受成为计算学科所带来的教育后果。

根据我自己的经验，我遇到过一些研究人员和工程师，他们能够处理复杂的机器人问题，却不理解构建软件的基本机制：编译做什么、链接做什么、为什么链接器找不到符号、什么是共享库，或者像 CMake 这样的构建系统实际上在做什么。这与了解编译器理论或能够编写链接器无关。问题在于缺乏关于源代码如何变成运行程序的基本心智模型。

同样的问题出现在网络、操作系统、权限、进程和安全领域。远程访问服务可能被随意暴露，因为访问机器人往往被当作最终目标，而关于认证、网络边界或还有谁可能到达该服务的问题则被视为次要问题。权限有时被绕过而不是被理解。网络问题的调试方式是反复更改设置直到某样东西能工作，而不是通过推理路由、端口、接口或信任边界。

这些并不是计算机科学中的冷门领域。它们是现代机器人所依赖的机器的基本属性。

这一点尤为重要，因为机器人不仅仅是一个附着在电机上的算法。现代机器人是一台联网的计算机系统，上面附着执行器。它可能包含多台计算机、摄像头、麦克风、无线接口、远程管理服务、GPU、嵌入式控制器、云连接以及来自许多不同厂商的软件。普通桌面应用程序中的错误可能导致应用程序崩溃。机器人系统中的错误可能暴露摄像头、提供机器的远程控制，或影响在物理世界中移动的东西。

Yarbo 机器人割草机中发现的安全问题就是一个例子。研究人员发现了涉及远程访问和特权凭据的严重弱点。这类失败之所以引人注目，并不是因为理解它们需要某种艰深的密码学突破：Yarbo 有意构建了远程管理访问，但显然其安全模型设计得极其糟糕。

人们很容易将这类事件简单地描述为网络安全失败，但这可能过于狭隘了。它们也可能揭示出更根本的计算机系统知识的缺乏——或者一个未能确保这种知识存在于设计和审查过程中的组织。不需要成为专业安全研究人员也能理解，同一个 root 密码不应该硬编码在所有部署的机器上、可远程访问的控制接口需要仔细的认证和隔离。

组织层面的问题同样重要。在机器人研究小组和初创公司中，工程师可能因为计算机视觉、控制、机械设计或机器学习方面的专长而被聘用，然后逐渐因为“总得有人来做”而负责软件部署、网络、远程访问、构建基础设施或系统管理。某一技术领域的专长被悄然视为意味着所有其他领域的能力。

恰恰相反。

知道如何训练视觉模型与知道如何构建计算机系统不是一回事。懂控制理论并不意味着理解操作系统权限。嵌入式电子方面的专长并不自动意味着网络安全的胜任能力。这些都是不同的知识体系，而机器人技术越来越需要它们相互作用。

ROS 既回应了这种状况，也强化了这种状况。

ROS 非常有价值，正是因为它隐藏了大量的系统复杂性。它提供了通信抽象、包约定、启动机制、工具和通用结构，使研究人员能够专注于机器人问题，而不是反复重建基础设施。

这是好的工程。

然而，当抽象成为理解的替代品，而不是建立在其上的工具时，问题就开始了。研究人员可能学会运行哪个 ROS 命令，却不理解正在创建什么进程、包是如何构建的、库是如何找到的、机器之间如何通信，或者为什么更改一个环境变量突然就修复了系统。

只要抽象正常工作，这看起来完全够用。

然而，机器人技术是一个抽象层经常失效的领域。驱动失败，设备消失，网络变得不可靠，库冲突，时序变化，GPU 内存耗尽，进程崩溃，权限阻止硬件访问，针对一个库版本构建的二进制文件被部署到另一台机器上——研究人员可能仅仅把构建好的二进制文件复制粘贴到另一台机器上后，就困惑于为什么“它不工作了”。突然间，研究人员跨越了“使用 ROS”和“理解计算机”之间的界限。

因此，问题不在于抽象本身。好的抽象是现代机器人技术得以可能的原因之一。也不应该期望每个机器人研究人员都成为操作系统专家、网络工程师和编译器开发者。

但在抽象之下，必须有一个最低限度的系统素养水平。

一个将软件部署到联网物理机器上的人，至少应该在概念上理解进程、用户和权限、网络、编译和链接、库、依赖关系以及基本的安全边界。这些不再是机器人技术的边缘技能。它们是理解实验装置本身的一部分。

然而，机器人研究的激励机制往往朝着相反的方向推动。研究人员因新的控制器、感知算法、规划方法或基准测试结果而获得奖励。花时间理解链接器问题、改进部署基础设施、审查远程访问安全性或提升构建系统的鲁棒性，很少能产出一篇论文。

因此，理性的反应是只学习足够的计算知识来让实验运行起来。

ROS、Ubuntu、Docker、Python 环境和日益复杂的工具使这比以往任何时候都更容易。这是有用的，但这也意味着一个人可以在软件栈的相当高的层次上操作，却对下面的东西只有一个模糊的概念。

结果是一个奇怪的矛盾：机器人技术是最以系统为导向的工程领域之一，但它的研究文化却允许人们在极其复杂的系统上工作，而不要求他们理解位于这些系统核心的计算机。

当一切正常时，这些知识看起来可能是多余的。

当出现不正常时，机器人会迅速提醒我们事实并非如此。

当一切都变成 `pip install`

这种“只学够让实验跑起来就行”的压力，塑造了现代机器人技术的工具文化。Python、Conda、预编译 wheel、容器以及越来越完整的软件环境，都减少了从想法到可用结果所需的系统知识量。

这是它们最大的优势之一。

也是下一个问题的起点。

Python 在机器人研究中占据主导地位是有充分理由的。它缩短了想法与实验之间的距离。但围绕它的工作流程越来越鼓励用户将软件视为可以简单地安装、导入和执行的东西，而底层的原生库、二进制接口、编译器工具链、驱动程序和运行时依赖则从视野中消失。

像

pip install torch

这样的命令看起来很简单，但它安装的不仅仅是“Python 代码”。像 PyTorch 这样的包严重依赖编译过的 C/C++ 和 CUDA 组件。当前的 PyTorch 二进制文件可能自带 CUDA 运行时库，而不是直接依赖系统 CUDA 工具包。

一旦 Python 世界与其外部的软件交互，这就变得混乱了。

用户可能有一个包含 PyTorch 及其 CUDA 运行时的 Conda 环境，同时又在使用针对主机系统上 CUDA 安装构建的 C++ 代码。不同的进程可以合法地使用不同的 CUDA 运行时，但当原生扩展、插件、构建环境或运行时搜索路径跨越这些边界时，问题就开始了。哪个库被加载，可能取决于搜索路径、RPATH、环境变量以及库的加载顺序。

到了这个时候，“我使用的是哪个 CUDA 版本？”就不再有一个显而易见的答案了。

在 Conda 环境内构建 Python wheel 时也会出现同样的问题。一个包可能编译一个原生扩展，并悄无声息地从主机系统发现头文件或共享库。生成的 wheel 在开发者的机器上能工作，但它实际上并不是完全独立的：它意外地获得了对环境外部库的依赖。后来，另一个版本的同一个库可能出现在 Conda 内部，环境可能被移到另一台计算机上，或者系统库可能发生变化。

于是，失败之所以看起来神秘，只是因为依赖边界从一开始就未被理解。

这是现代 Python 科学生态系统的奇怪属性之一：它给人一种高层次的、自包含的表象，但其底层却建立在一个日益复杂的原生软件集合之上。Python 包管理器在包层面上解决依赖问题，但原生软件最终仍然生活在编译器、ABI、共享对象、动态加载器、驱动程序和操作系统搜索路径的世界里。

这些世界并不总是能干净地对齐。

更糟糕的是，许多用户根本不知道他们正在跨越那个边界。wheel 被体验为一个 Python 包，即使大部分重要工作是由编译库执行的。Conda 环境被体验为一个隔离的环境，即使在其中构建的软件已经链接到了主机。机器学习库被体验为一个 Python 库，即使它的执行依赖于 CUDA 用户空间库与主机 NVIDIA 驱动栈之间的特定关系。

当这变得足够困难时，标准答案越来越倾向于：

干脆用 Docker 就行了。

Docker 非常有用，尤其是在可复现性和部署方面。在机器人技术中，将不同组件作为独立镜像分发、隔离依赖、固定版本，并在许多机器上部署相同的软件，是有意义的。我甚至见过商业机器人交付时运行着几个 Docker 容器来承载系统的不同部分，这正在成为一种常见的架构模式。

但 Docker 也越来越被视为一种万能的解决方案，用来处理没有人愿意去理解的依赖问题。

容器可以使复杂的依赖图可复现，而不使该依赖图被理解。

GPU 的情况特别清楚地说明了这一点。包含 CUDA 软件的 Docker 镜像实际上并不独立于主机 GPU 栈。NVIDIA 内核驱动仍然在主机上，容器中的 GPU 访问依赖于额外的机制，如 NVIDIA Container Toolkit。因此，所谓自包含的容器仍然依赖于主机驱动、设备节点、容器运行时、CUDA 用户空间库和容器内应用程序之间精心维护的边界。

这个抽象大致是这样的：

主机 Linux 内核
    ↓
主机 NVIDIA 驱动
    ↓
NVIDIA Container Toolkit
    ↓
容器运行时
    ↓
容器内的 CUDA 库
    ↓
PyTorch / TensorRT / ROS
    ↓
机器人应用程序

当一切正常时，用户可能只看到最后一行。

当它停止工作时，主机驱动和 CUDA 运行时之间的区别突然就重要了。容器运行时配置也很重要。设备权限也很重要。GPU 设备传入容器的方式也很重要。

而机器人技术使容器边界变得更加不干净。

Web 服务通常可以舒适地运行在容器内，通过网络套接字通信。机器人则经常需要直接访问摄像头、串行设备、USB 设备、CAN 接口、GPU、共享内存、实时调度，有时还需要主机网络。每一项需求都在所谓隔离的环境上又打了一个洞。

容器化软件越接近物理硬件，完全自包含容器的想法就越不可信。

同样，这并不意味着容器是糟糕的工程。恰恰相反：当边界被有意识地理解和设计时，它们可以成为部署复杂机器人系统的最佳方式之一。

问题在于将它们用作理解这些边界的替代品。

有一个反复出现的模式：

原生依赖变得困难
        ↓
使用 Python 环境

Python 和原生依赖变得困难
        ↓
使用 Conda

Conda 和系统依赖变得困难
        ↓
使用 Docker

Docker 和硬件集成变得困难
        ↓
添加运行时钩子、设备映射、主机网络、
特权、卷和厂商特定工具

依赖问题并没有消失。它只是转移了。

与此同时，栈上又增加了一层抽象，而操作它的人可能对其底层发生的事情了解得更少了。

这正是 Python 化工作流程值得关注的原因，不仅仅是语言偏好的问题。它反映了研究文化中更广泛的变革。主导目标越来越倾向于让软件易于消费：安装包，激活环境，启动容器，运行 launch 文件。

这极具生产力，研究激励机制也奖励这种做法。研究人员想要测试算法，而不是花三天时间理解 ELF 符号解析或 CUDA 库加载。

但当这种便利成为研究人员所知道的唯一计算模型时，代价就出现了。

Python 本身不是问题。Conda 不是问题。Docker 不是问题。ROS 不是问题。每一个都解决了一个真实而困难的问题。

但当每一个抽象被用来掩盖前一个抽象，而没有人对完整系统保持清晰的心智模型时，问题就出现了。

在某个时刻，机器人研究人员可能正在使用 Docker 容器内 Conda 环境中的 Python 代码，调用 C++ 和 CUDA 库，通过 ROS 通信，通过运行时特定机制访问 GPU 和硬件设备，运行在他们自己并不完全理解的 Ubuntu 系统之上。

一切可能都能工作。

但当它不工作时，调试问题就不再是 Python 问题、ROS 问题、Docker 问题或 Ubuntu 问题了。

它是一个计算机系统问题。

而这恰恰是这种工作流程最容易让人避免学习的知识。

结论：便利、抽象与控制权的丧失

本文讨论的技术本身都不是问题所在。Ubuntu、ROS、Python、Conda 和 Docker 之所以流行，是因为它们解决了真实的问题，使困难的工作变得更容易。

问题在于当便利变成依赖、抽象变成不透明、标准化变成集中化时会发生什么。

Ubuntu 有把“Linux”变成一个由单一厂商定义的平台的风险。ROS 强化了这个平台，因为不这样做成本太高。机器人文化往往更奖励让实验运行起来，而不是理解运行实验的系统。Python 化的工作流程和容器则使人们越来越有可能在栈的高层操作，而不必学习底层的任何东西。

每一步在个体上都是理性的。但它们共同产生了一个更容易使用却更难理解的生态系统，更容易标准化却更难摆脱，并且越来越依赖那些出故障后很少有人能调试的抽象。编码智能体（coding agent）可能通过使人们能够借助语言而非理解来修复系统，从而加速这一趋势。

也许真正的问题不在于这些工具是否给了我们足够的自由。

而在于当抽象停止工作时，我们是否还知道那种自由意味着什么。

本文由欧亚非撰写，ChatGPT 辅助完成（2026-08-15）。原文以英文撰写，并由 DeepSeek 辅助翻译（2026-08-16）；最终审阅与润色由欧亚非完成。
© 2026 Yafei Ou。保留所有权利。

Introducing CRESSim-Ultrasound

Sat, 15 Aug 2026 00:00:00 GMT

import ResponsiveImageGrid from "../../../../components/misc/ResponsiveImageGrid.astro"; import ImageUltrasoundRestTissue from "./ultrasound-rest-tissue.png"; import ImageUltrasoundDeformedTissue from "./ultrasound-deformed-tissue.png";

CRESSim-Ultrasound is a CUDA-only ultrasound simulation library. It is a lightweight reimplementation of OpenBCSim designed for GPU-based ultrasound simulation and can be easily integrated into robotics and simulation workflows.

The source code is available on GitHub:

::github{repo="yafei-ou/CRESSim-Ultrasound"}

What it provides

The library exposes a C API for constructing scan sequences, configuring a Gaussian excitation signal and beam profile, binding scatterers, and running the simulation. Its main compute interface keeps scatterers, scanlines, and RF output buffers on the GPU, avoiding unnecessary CPU-GPU transfers.

For applications in which the probe pose changes frequently, scanlines can be supplied from a device-resident buffer instead of rebuilding and uploading a CPU-side scan sequence. The simulator also supports CUDA streams and an external CUDA completion semaphore, which makes it practical to compose with other GPU workloads (such as Vulkan).

Deformation-aware scatterers

One useful extension is the deformation tracker. It generates dense ultrasound scatterers from sparse input points, records their initial configuration, and updates the scatterers as those input points move. This provides a direct path for coupling the ultrasound image formation pipeline to a deformable-tissue simulator (such as when using the material point method) while keeping the update on the GPU.

The current implementation uses a Gaussian beam profile. It is not yet a feature-complete replacement for OpenBCSim; in particular, lookup-table beam-profile support is not implemented.

Written by Yafei Ou, assisted by ChatGPT (2026-08-15).
© 2026 Yafei Ou. All rights reserved.

CRESSim-Ultrasound：基于 CUDA 的轻量级超声仿真库

Sat, 15 Aug 2026 00:00:00 GMT

CRESSim-Ultrasound 是一个基于 CUDA 实现的超声仿真库。它是 OpenBCSim 的轻量级重新实现，面向 GPU 超声仿真，并且可以方便地集成到机器人和仿真工作流中。

源代码已在 GitHub 开源：

::github{repo="yafei-ou/CRESSim-Ultrasound"}

主要功能

该库提供 C 语言应用程序接口（C API），用于构建扫描序列、配置高斯激励信号和高斯声束轮廓、绑定散射体，以及运行仿真。其主要计算接口将散射体、扫描线和射频（RF）输出缓冲区保留在 GPU 上，从而避免不必要的主机端（CPU）与 GPU 之间的数据传输。

对于探头位姿频繁变化的应用，扫描线可直接由设备端缓冲区提供，无需重新构建并上传主机端的扫描序列。模拟器还支持 CUDA 流和用于完成通知的外部 CUDA 信号量，因此可与其他 GPU 工作负载（如 Vulkan）组合使用。

支持形变的散射体

一个实用的扩展功能是形变跟踪器。它可以根据稀疏输入点生成稠密的超声散射体，记录散射体的参考构形，并在输入点移动时更新散射体位置。这为超声成像流程与可形变组织模拟器（例如基于物质点法的模拟器）在 GPU 上直接耦合提供了途径。

当前的代码实现使用高斯声束轮廓。它尚不是 OpenBCSim 的完整替代品；尤其是尚未实现基于查找表（LUT）的声束轮廓支持。

作者：欧亚非与 ChatGPT（2026-08-15）。原文以英文撰写，并在 ChatGPT 的协助下翻译；最终审阅与润色由欧亚非完成。
© 2026 欧亚非。保留所有权利。

Simulation of Surgical Suturing Using Position-Based Dynamics and the Material Point Method for Robot Reinforcement Learning

Wed, 29 Jul 2026 00:00:00 GMT

Full Article

Simulation of Surgical Suturing Using Position-Based Dynamics and the Material Point Method for Robot Reinforcement Learning
Tleukhan Mussin, Yafei Ou, Mahdi Tavakoli
IEEE RAS/EMBS 11th International Conference on Biomedical Robotics and Biomechatronics (BioRob), 2026

arXiv: https://arxiv.org/abs/2607.27494

Abstract

Recent advances in robotics research have created a strong demand for high-performance simulators. Surgical robotics simulation faces unique challenges due to the need to model diverse objects, such as rigid instruments, soft tissue, and fluids. While many studies simulate sutures or soft tissue independently, only a few have considered the complete soft-tissue suturing scenario, including the contact between sutures and deformable tissue during suture insertion. Building on previous work, this paper presents a novel suturing simulation environment using sutures modelled by position-based dynamics (PBD) and soft bodies modelled by the material point method (MPM) while considering two-way contact with frictional and drag forces. We introduce a contact coupling method between the PBD suture and the MPM soft tissue, enabling visually plausible suture-tissue interactions. The simulator is optimized for GPU execution with parallel scenes using multiple CUDA streams, and we present a Reinforcement Learning (RL) environment for autonomous suturing sub-tasks, including needle insertion, driving, and extraction. Using ML-Agents, RL agents trained in the simulator show stable learning and achieve 80% and 68% success rates in needle insertion and extraction, respectively, under the strictest distance threshold.

Citation

@misc{mussin2026simulation,
  title={Simulation of Surgical Suturing Using Position-Based Dynamics and the Material Point Method for Robot Reinforcement Learning},
  author={Mussin, Tleukhan and Ou, Yafei and Tavakoli, Mahdi},
  year={2026},
  eprint={2607.27494},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2607.27494}
}

A Quick Hack for Texturing a 3D Model

Mon, 16 Mar 2026 00:00:00 GMT

import ImageWrapper from "../../../../components/misc/MarkdownImageWrapper.astro"; import ImageRealSenseGenerated from "./realsense-generated.png"; import ImageRealSense from "./realsense-d435i.png"; import ImageRealSense2 from "./realsense-d435i-2.png";

Although the CAD model (SolidWorks) for the RealSense D435i camera is available in the official documentation, I could not find a textured 3D model suitable for simulation or rendering.

So I tried a quick workaround: generating views of each side of the camera using images of the device found online, with the help of Nano Banana Pro.

See one of the generated views below:

These generated views can then be used as textures during UV unwrapping (e.g., using Project from View). By simply aligning the vertices with the image, the model can quickly obtain a usable texture.

This is far from perfect, but given the minimal cost (except for the Google One subscription fee), the results are surprisingly usable.

Written by Yafei Ou and reviewed for grammar and conceptual accuracy by ChatGPT (2026-03-16).
© 2026 Yafei Ou. All rights reserved.

一个快速为 3D 模型添加纹理的小技巧

Mon, 16 Mar 2026 00:00:00 GMT

import ImageWrapper from "../../../../components/misc/MarkdownImageWrapper.astro"; import ImageRealSenseGenerated from "./realsense-generated.png"; import ImageRealSense from "./realsense-d435i-2.png"; import ImageRealSense2 from "./realsense-d435i.png";

尽管在官方文档中可以找到 RealSense D435i 相机的 CAD 模型（SolidWorks），但是我没找到能够用于仿真或渲染的带纹理三维模型。

因此，我尝试了一个简单的变通方法：借助 Nano Banana Pro，利用网上找到的设备图片生成相机各个侧面的视图。

下面是生成视图中的一个示例：

这些生成的视图可以在进行 UV 展开时用作纹理（例如使用“从视图投影”）。只需要将模型的顶点与图像对齐，就可以快速为模型赋予一个可用的纹理。

虽说远非完美，但考虑到几乎不费什么成本（除了 Google One 订阅费），已经相当实用。

CRESSim–MPM: A Material Point Method Library for Surgical Soft Body Simulation with Cutting and Suturing

Sun, 19 Oct 2025 00:00:00 GMT

import YouTubeVideo from '../../../components/misc/YouTubeVideo.astro';

Full Article

CRESSim-MPM: A Material Point Method Library for Surgical Soft Body Simulation with Cutting and Suturing
Yafei Ou, Mahdi Tavakoli
2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

IEEE Xplore: https://ieeexplore.ieee.org/document/11247388
arXiv: https://arxiv.org/abs/2502.18437

Abstract

A number of recent studies have focused on developing surgical simulation platforms to train machine learning (ML) agents or models with synthetic data for surgical assistance. While existing platforms excel at tasks such as rigid body manipulation and soft body deformation, they struggle to simulate more complex soft body behaviors like cutting and suturing. A key challenge lies in modeling soft body fracture and splitting using the finite-element method (FEM), which is the predominant approach in current platforms. Additionally, the two-way suture needle/thread contact inside a soft body is further complicated when using FEM. In this work, we use the material point method (MPM) for such challenging simulations and propose new rigid geometries and soft-rigid contact methods specifically designed for them. We introduce CRESSim-MPM, a GPU-accelerated MPM library that integrates multiple MPM solvers and incorporates surgical geometries for cutting and suturing, serving as a specialized physics engine for surgical applications. It is further integrated into Unity, requiring minimal modifications to existing projects for soft body simulation. We demonstrate the simulator's capabilities in real-time simulation of cutting and suturing on soft tissue and provide an initial performance evaluation of different MPM solvers when simulating varying numbers of particles.

Code

CUDA/C++ library

::github{repo="yafei-ou/cressim-mpm"}

Citation

@article{ou2025cressim,
  title={CRESSim-MPM: A Material Point Method Library for Surgical Soft Body Simulation with Cutting and Suturing},
  author={Ou, Yafei and Tavakoli, Mahdi},
  journal={arXiv preprint arXiv:2502.18437},
  year={2025}
}

Learning Autonomous Surgical Irrigation and Suction with the da Vinci Research Kit Using Reinforcement Learning

Wed, 04 Jun 2025 00:00:00 GMT

import YouTubeVideo from '../../../components/misc/YouTubeVideo.astro';

Full Article

Learning Autonomous Surgical Irrigation and Suction with the da Vinci Research Kit Using Reinforcement Learning
Yafei Ou, Mahdi Tavakoli
IEEE Transactions on Automation Science and Engineering, vol. 22, pp. 16753-16767, 2025

IEEE Xplore: https://ieeexplore.ieee.org/abstract/document/11023631
arXiv: https://arxiv.org/abs/2411.14622

Abstract

The irrigation-suction process is a common procedure to rinse and clean up the surgical field in minimally invasive surgery (MIS). In this process, surgeons first irrigate liquid, typically saline, into the surgical scene for rinsing and diluting the contaminant, and then suction the liquid out of the surgical field. While recent advances have shown promising results in the application of reinforcement learning (RL) for automating surgical subtasks, fewer studies have explored the automation of fluid-related tasks. In this work, we explore the automation of both steps in the irrigation-suction procedure and train two vision-based RL agents to complete irrigation and suction autonomously. To achieve this, a platform is developed for creating simulated surgical robot learning environments and for training agents, and two simulated learning environments are built for irrigation and suction with visually plausible fluid rendering capabilities. With techniques such as domain randomization (DR) and carefully designed reward functions, two agents are trained in the simulator and transferred to the real world. Individual evaluations of both agents show satisfactory real-world results. With an initial amount of around 5 grams of contaminants, the irrigation agent ultimately achieved an average of 2.21 grams remaining after a manual suction. As a comparison, fully manual operation by a human results in 1.90 grams remaining. The suction agent achieved 2.64 and 2.24 grams of liquid remaining across two trial groups with more than 20 and 30 grams of initial liquid in the container. Fully autonomous irrigation-suction trials reduce the contaminant in the container from around 5 grams to an average of 2.42 grams, although yielding a higher total weight remaining (4.40) due to residual liquid not suctioned. Further information about the project is available at this https URL.

Code

CRESSim (Unity Project)

::github{repo="tbs-ualberta/cressim"}

PhysX 5 for Unity

::github{repo="yafei-ou/physx5-for-unity"}

PhysX 5 for Unity Native Plugin (C++)

::github{repo="yafei-ou/physx5-native-plugin"}

Citation

@article{ou2025learning,
  title={Learning autonomous surgical irrigation and suction with the da vinci research kit using reinforcement learning},
  author={Ou, Yafei and Tavakoli, Mahdi},
  journal={IEEE Transactions on Automation Science and Engineering},
  year={2025},
  volume={22},
  pages={16753-16767},
  publisher={IEEE}
}

Thoughts on Modern C++

Sat, 17 May 2025 00:00:00 GMT

Modern C++ is undeniably powerful. But, as with any programming language, it is also often hated by many. And there are certainly reasons.

The Legacy

I have been using C++ for around a year for some projects, but without formal training or specific tutorials. I really started learning it by reading Nvidia's PhysX codebase, which I found very well-organized and performance-driven. PhysX has been used by many games--as well as game engines like Unity. On the other hand, the codebase is far from being "modern" from many C++ users' view. STL is not even used--no smart pointers, no std::vector, all implemented within PhysX in an old fashioned way.

It is not difficult to make some guesses as to the reasons:

PhysX was initially developed around 20 years ago when the standard hadn't even introduced the modern features;
PhysX really wants to squeeze that last bit of performance, especially on low-end devices;
PhysX should support those devices that your regular program won't run on, e.g. Nintendo Switch (NS). I don't even know how much of the standard Nintendo has implemented for the NS, as it is completely proprietary.

I am not sure how each of these points has contributed to this design, and there are likely other factors influencing PhysX’s architecture. But here's the practical question: should we always use modern C++ features when starting a new project now? I do not have the answer myself, but since I learned C++ by reading PhysX's codebase, I have stayed away from many modern features.

The Confusing Standard

I do like many of the modern additions and changes to the C++ standard, which simplify and unify many aspects. Still, the standard’s wording sometimes leaves even experienced users confused. I once blamed my own skill level until I read P1839R7 and realized ambiguities (and possibly defects) in the standard are a common headache. Below is quoted from the paper:

void print_hex(int n) {
    unsigned char* a = (unsigned char*)(&n);
    for (int i = 0; i < sizeof(int); ++i)
        printf("%02x ", a[i]);
}

int main() {
    print_hex(123456);
}

In C, this is a valid program. On a little-endian machine where sizeof(int) == 4, this will print 40 e2 01 00. In C++, this is widely assumed to be valid as well, and this functionality is widely used in existing code bases (think of binary file formats, hex viewers, and many other low-level use cases).

However, surprisingly, in C++ this code has undefined behaviour under the current specification. In fact, it is impossible in C++ to directly access the object representation of an object (i.e. to read its underlying bytes), even for built-in types such as int. Instead, we would have to use memcpy to copy the bytes into a separate array of unsigned char, and access them from there. However, this workaround only works for trivially copyable types. It also directly violates one of the fundamental principles of C++: to leave no room for a lower-level language.

The goal of this paper is to provide the necessary wording fixes to make accessing object representations such as in the code above defined behaviour. Existing compilers already assume that this should be valid. The goal of the paper is therefore to not require any changes to existing compilers or existing code, but to legalise existing code that already works in practice and was always intended to be valid.

Practically, compiler implementations and their documentation are the de facto standard, rather than the formal C++ standard. One example is union type punning.

#include <iostream>
#include <cstdint>

union Pun {
    std::uint32_t i;
    float         f;
};

int main() {
    Pun u;
    u.i = 0x3F800000;      // Bit pattern for 1.0f in IEEE-754
    std::cout << u.f      // Reads the float interpretation
              << std::endl;
    return 0;
}

The use case is well-defined by the C standard, but is UB in C++. The C++ standard does not define the behavior (likely deliberately) and leaves more freedom for compilers' implementation. Practically I don't see any compiler that does not support this. But it is very confusing to look up and be certain. GCC states this explicitly:

The practice of reading from a different union member than the one most recently written to (called “type-punning”) is common. Even with -fstrict-aliasing, type-punning is allowed, provided the memory is accessed through the union type. So, the code above works as expected. See Structures, Unions, Enumerations, and Bit-Fields.

However, I can't find any information about MSVC's treatment on this case, although it is obviously supported.

The ambiguity on the wording of the standard, either by design or by oversight, makes it almost a common practice to wait for at least 5 years before a standard can be used until things stabilize. There seems to be no hope for improvements in the situation.

How About CUDA?

CUDA's rise for GPGPU, and especially with recent hype in machine learning, has attracted another unique group of C++ users. NVCC's implementation of C++ is very unique, as the code is compiled for the GPU device instead of the CPU. This adds an additional layer of confusion.

One example is pointer casting. Using this for type punning is UB that violates the strict aliasing rule (although at least MSVC shouldn't have trouble with this). It is usually more proper to use union or memcpy to achieve this (although union doesn't seem portable as per the standard), but Nvidia's official documentation implies that at least this will definitely work for their GPUs. Below is taken from the documentation:

#if __CUDA_ARCH__ < 600
__device__ double atomicAdd(double* address, double val)
{
    unsigned long long int* address_as_ull =
                              (unsigned long long int*)address;
    unsigned long long int old = *address_as_ull, assumed;

    do {
        assumed = old;
        old = atomicCAS(address_as_ull, assumed,
                        __double_as_longlong(val +
                               __longlong_as_double(assumed)));

    // Note: uses integer comparison to avoid hang in case of NaN (since NaN != NaN)
    } while (assumed != old);

    return __longlong_as_double(old);
}
#endif

Casting from a double* to a unsigned long long int* and dereferencing it is obviously UB. But this code snippet suggests that it should be well-defined, at least for device code.

Conclusion

My experience with developing a cross-platform C++/CUDA project has been painful, and the aspects mentioned above have been lingering in my mind since the beginning.

There has not been a practical alternative to C++ at least for CUDA programming (or even broadly, most GPGPU programming). I guess the only way is to live with it, live with a confusing standard, and live with UBs in my code.

Written by Yafei Ou and reviewed for grammar and conceptual accuracy by ChatGPT (o4-mini-high).
© 2025 Yafei Ou. All rights reserved.

Autonomous Soft-Tissue Needle Steering Using Reinforcement Learning Guided by Human Input

Tue, 24 Dec 2024 00:00:00 GMT

import YouTubeVideo from '../../../components/misc/YouTubeVideo.astro';

Full Article

Autonomous Soft-Tissue Needle Steering Using Reinforcement Learning Guided by Human Input
Yafei Ou, Mahdi Tavakoli
Journal of Medical Robotics Research, vol. 09, no. 03n04, 2024.

Link: https://www.worldscientific.com/doi/abs/10.1142/S2424905X24400105

Abstract

Soft-tissue needle steering, where a deformable needle is inserted into the tissue to guide its tip to a desired position, is a common minimally invasive surgery (MIS) procedure. The diverse types of needles and complex tissue dynamics limit the use of existing approaches that utilize models of the needle and the tissue for automating the task. In this work, we employ a data-driven approach using deep reinforcement learning (DRL) to achieve autonomous needle steering by viewing it as a multi-goal reinforcement learning problem. Human interventions are incorporated during training to accelerate learning and reduce catastrophic failures. Generative adversarial imitation learning (GAIL) is combined with regular DRL by utilizing a hindsight relabeling scheme for human interventions to encourage the agent to imitate human behavior. To emulate the sim-to-real process, an agent is first trained in a simplistic simulation environment for needle steering and then transferred to a sophisticated one considered as the real world with fine-tuning (sim-to-sim). Experimental results show that with human interventions, the proposed method outperforms the other compared DRL approaches and can achieve good performance with only 2,000 training steps in the complex simulation environment, achieving an average return comparable to that of a 55,000-step agent trained from scratch.

Citation

@article{ou2024autonomous,
  title={Autonomous Soft-Tissue Needle Steering Using Reinforcement Learning Guided by Human Input},
  author={Ou, Yafei and Tavakoli, Mahdi},
  journal={Journal of Medical Robotics Research},
  volume={9},
  number={03n04},
  pages={2440010},
  year={2024},
  publisher={World Scientific}
}

Autonomous Blood Suction for Robot-Assisted Surgery: A Sim-to-Real Reinforcement Learning Approach

Mon, 01 Jul 2024 00:00:00 GMT

import YouTubeVideo from '../../../components/misc/YouTubeVideo.astro';

Full Article

Autonomous Blood Suction for Robot-Assisted Surgery: A Sim-to-Real Reinforcement Learning Approach
Yafei Ou, Abed Soleymani, Xingyu Li, Mahdi Tavakoli
IEEE Robotics and Automation Letters, vol. 9, no. 8, pp. 7246-7253, Aug. 2024

IEEE Xplore: https://ieeexplore.ieee.org/abstract/document/10578312

Abstract

Recent applications of deep reinforcement learning (DRL) in surgical autonomy have shown promising results in automating various surgical sub-tasks. While most of these studies consider the rigid and soft body dynamics in the surgery such as tissue deformation, only a few have investigated the situation where fluid is present. However, the presence of fluids, particularly blood, is common in surgeries, and interacting with them adds additional challenges to task automation. In this work, we investigate the use of DRL in automating blood suction, a common surgical sub-task where blood is removed from the surgical field. We build a blood suction simulation environment based on position-based fluids (PBF), train an agent with domain-randomized environment parameters through curriculum learning, and obtain a generalizable policy that can be applied to various shapes of tissue and types of liquid. Real-world experiments show that the agent can perform autonomous suction in different tissue models with different amounts and types of liquid, and only one of the 50 trials resulted in more than 3 ml of blood remaining.

Citation

@article{ou2024autonomous,
  title={Autonomous Blood Suction for Robot-Assisted Surgery: A Sim-to-Real Reinforcement Learning Approach},
  author={Ou, Yafei and Soleymani, Abed and Li, Xingyu and Tavakoli, Mahdi},
  journal={IEEE Robotics and Automation Letters},
  year={2024},
  publisher={IEEE}
}

A Realistic Surgical Simulator for Non-Rigid and Contact-Rich Manipulation in Surgeries with the da Vinci Research Kit

Mon, 08 Apr 2024 00:00:00 GMT

import YouTubeVideo from '../../../components/misc/YouTubeVideo.astro';

Full Article

A Realistic Surgical Simulator for Non-Rigid and Contact-Rich Manipulation in Surgeries with the da Vinci Research Kit
Yafei Ou, Sadra Zargarzadeh, Paniz Sedighi, Mahdi Tavakoli
2024 21st International Conference on Ubiquitous Robots (UR)

IEEE Xplore: https://ieeexplore.ieee.org/document/10597513
arXiv: https://arxiv.org/abs/2404.05888

Abstract

Realistic real-time surgical simulators play an increasingly important role in surgical robotics research, such as surgical robot learning and automation, and surgical skills assessment. Although there are a number of existing surgical simulators for research, they generally lack the ability to simulate the diverse types of objects and contact-rich manipulation tasks typically present in surgeries, such as tissue cutting and blood suction. In this work, we introduce CRESSim, a realistic surgical simulator based on PhysX 5 for the da Vinci Research Kit (dVRK) that enables simulating various contact-rich surgical tasks involving different surgical instruments, soft tissue, and body fluids. The real-world dVRK console and the master tool manipulator (MTM) robots are incorporated into the system to allow for teleoperation through virtual reality (VR). To showcase the advantages and potentials of the simulator, we present three examples of surgical tasks, including tissue grasping and deformation, blood suction, and tissue cutting. These tasks are performed using the simulated surgical instruments, including the large needle driver, suction irrigator, and curved scissor, through VR-based teleoperation.

Code

CRESSim (Unity Project)

::github{repo="tbs-ualberta/cressim"}

PhysX 5 for Unity

::github{repo="yafei-ou/physx5-for-unity"}

PhysX 5 for Unity Native Plugin (C++)

::github{repo="yafei-ou/physx5-native-plugin"}

Citation

@inproceedings{ou2024realistic,
  title={A Realistic Surgical Simulator for Non-Rigid and Contact-Rich Manipulation in Surgeries with the da Vinci Research Kit}, 
  author={Ou, Yafei and Zargarzadeh, Sadra and Sedighi, Paniz and Tavakoli, Mahdi},
  booktitle={2024 21st International Conference on Ubiquitous Robots (UR)}, 
  year={2024},
  pages={64-70}
}

Towards Safe and Efficient Reinforcement Learning for Surgical Robots Using Real-Time Human Supervision and Demonstration

Wed, 19 Apr 2023 00:00:00 GMT

import YouTubeVideo from '../../../components/misc/YouTubeVideo.astro';

Full Article

Towards Safe and Efficient Reinforcement Learning for Surgical Robots Using Real-Time Human Supervision and Demonstration
Yafei Ou, Mahdi Tavakoli
2023 International Symposium on Medical Robotics (ISMR)

IEEE Xplore: https://ieeexplore.ieee.org/abstract/document/10130214

Abstract

Recent research in surgical robotics has focused on increasing the level of autonomy in order to reduce the workload of surgeons. While deep reinforcement learning (DRL) has shown promising results in automating some surgical subtasks, due to its demand for a large number of random explorations, safety and learning efficiency remain the primary challenges when applying it to surgical robot learning. In this work, we present a DRL framework with real-time human supervision during the training process for surgical robot learning to avoid significant failures and speed up training. A novel training methodology based on the combination of DRL and generative adversarial imitation learning (GAIL) is proposed to further improve learning efficiency by imitating human behaviors. The proposed method is validated using two simulated environments, where human intervention is performed via teleoperation. Results show that our method outperforms baseline algorithms and can achieve safe and efficient learning.

Citation

@inproceedings{ou2023towards,
  title={Towards safe and efficient reinforcement learning for surgical robots using real-time human supervision and demonstration},
  author={Ou, Yafei and Tavakoli, Mahdi},
  booktitle={2023 International Symposium on Medical Robotics (ISMR)},
  pages={1--7},
  year={2023},
  organization={IEEE}
}

Robot Learning Incorporating Human Interventions in the Real World for Autonomous Surgical Endoscopic Camera Control

Fri, 10 Mar 2023 00:00:00 GMT

import YouTubeVideo from '../../../components/misc/YouTubeVideo.astro';

Full Article

Robot Learning Incorporating Human Interventions in the Real World for Autonomous Surgical Endoscopic Camera Control
Yafei Ou, Mahdi Tavakoli
Journal of Medical Robotics Research, vol. 08, no. 03n04, p. 2340004, 2023.

Link: https://www.worldscientific.com/doi/abs/10.1142/S2424905X23400044

Abstract

Recent studies in surgical robotics have focused on automating common surgical subtasks such as grasping and manipulation using deep reinforcement learning (DRL). In this work, we consider surgical endoscopic camera control for object tracking e.g. using the endoscopic camera manipulator (ECM) from the da Vinci Research Kit (dVRK) (Intuitive Inc., Sunnyvale, CA, USA) as a typical surgical robot learning task. A DRL policy for controlling the robot joint space movements is first trained in a simulation environment and then continues the learning in the real world. To speed up training and avoid significant failures (in this case, losing view of the object), human interventions are incorporated into the training process and regular DRL is combined with generative adversarial imitation learning (GAIL) to encourage imitating human behaviors. Experiments show that an average reward of 159.8 can be achieved within 1000 steps compared to only 121.8 without human interventions, and the view of the moving object is lost only twice during the training process out of 3 trials. These results show that human interventions can improve learning speed and significantly reduce failures during the training process.

Citation

@article{ou2023robot,
  title={Robot learning incorporating human interventions in the real world for autonomous surgical endoscopic camera control},
  author={Ou, Yafei and Zargarzadeh, Sadra and Tavakoli, Mahdi},
  journal={Journal of Medical Robotics Research},
  volume={8},
  number={03n04},
  pages={2340004},
  year={2023},
  publisher={World Scientific}
}

Sim-to-Real Surgical Robot Learning and Autonomous Planning for Internal Tissue Points Manipulation Using Reinforcement Learning

Fri, 10 Mar 2023 00:00:00 GMT

import YouTubeVideo from '../../../components/misc/YouTubeVideo.astro';

Full Article

Sim-to-Real Surgical Robot Learning and Autonomous Planning for Internal Tissue Points Manipulation Using Reinforcement Learning
Yafei Ou, Mahdi Tavakoli
IEEE Robotics and Automation Letters, vol. 8, no. 5, pp. 2502-2509, May 2023

IEEE Xplore: https://ieeexplore.ieee.org/abstract/document/10065553

Abstract

Indirect simultaneous positioning (ISP), where internal tissue points are placed at desired locations indirectly through the manipulation of boundary points, is a type of subtask frequently performed in robotic surgeries. Although challenging due to complex tissue dynamics, automating the task can potentially reduce the workload of surgeons. This letter presents a sim-to-real framework for learning to automate the task without interacting with a real environment, and for planning preoperatively to find the grasping points that minimize local tissue deformation. A control policy is learned using deep reinforcement learning (DRL) in the FEM-based simulation environment and transferred to real-world situation. Grasping points are planned in the simulator by utilizing the trained policy using Bayesian optimization (BO). Inconsistent simulation performance is overcome by formulating the problem as a state augmented Markov decision process (MDP). Experimental results show that the learned policy places the internal tissue points accurately, and that the planned grasping points yield small tissue deformation among the trials. The proposed learning and planning scheme is able to automate internal tissue point manipulation in surgeries and has the potential to be generalized to complex surgical scenarios.

Citation

@article{ou2024autonomous,
  title={Autonomous Soft-Tissue Needle Steering Using Reinforcement Learning Guided by Human Input},
  author={Ou, Yafei and Tavakoli, Mahdi},
  journal={Journal of Medical Robotics Research},
  volume={9},
  number={03n04},
  pages={2440010},
  year={2024},
  publisher={World Scientific}
}

Site Changelog

Sat, 01 Jan 2022 00:00:00 GMT

The changelog of the site. The published date is set to be old enough so that this post will always appear at the bottom.

2026-08-15

Migrated to Astro 7

2026-01-22

Added astrojs/mdx and use MDX for most of the posts
Added some other publications
Increased cover image height

2025-05-18

Fixed an issue due to some characters in path (in tags)
Added the 404 page from Fuwari

2025-05-17

Migrated to Fuwari
Added publications and CV pages
Added posts on publications

2024-09-07

Updated the CV page and the PDF version.

2024-08-31

Updated the CV page and the PDF version.
Changed the website title to "Yafei's Archive"

2024-07-28

Added CV page
Removed redundant pages
Added Publications page (only one paper has been added)

Yafei's Archive

Introducing CRESSim-Neo: A GPU-Accelerated Simulation Engine for Surgical Robotics

What CRESSim-Neo is

Why build another simulator for surgical robotics?

What CRESSim-Neo enables

Source code and contributing

Citation

Ghost: 2 Big To Rig - Some Quick Thoughts

The Cost of Convenience: Ubuntu, Robotics, and the Loss of Control

Is Ubuntu Becoming Another Windows?

Roboticists Who Don't Understand Their Computers

When Everything Becomes pip install

Conclusion: Convenience, Abstraction, and the Loss of Control

便利的代价：Ubuntu、机器人技术与控制权的丧失

Ubuntu 正在变成另一个 Windows 吗？

不懂自己计算机的机器人研究者

当一切都变成 pip install

结论：便利、抽象与控制权的丧失

Introducing CRESSim-Ultrasound

What it provides

Deformation-aware scatterers

CRESSim-Ultrasound：基于 CUDA 的轻量级超声仿真库

主要功能

支持形变的散射体

Simulation of Surgical Suturing Using Position-Based Dynamics and the Material Point Method for Robot Reinforcement Learning

Full Article

Abstract

Citation

A Quick Hack for Texturing a 3D Model

一个快速为 3D 模型添加纹理的小技巧

CRESSim–MPM: A Material Point Method Library for Surgical Soft Body Simulation with Cutting and Suturing

Full Article

Abstract

Code

CUDA/C++ library

Citation

Learning Autonomous Surgical Irrigation and Suction with the da Vinci Research Kit Using Reinforcement Learning

Full Article

Abstract

Code

CRESSim (Unity Project)

PhysX 5 for Unity

PhysX 5 for Unity Native Plugin (C++)

Citation

Thoughts on Modern C++

The Legacy

The Confusing Standard

How About CUDA?

Conclusion

Autonomous Soft-Tissue Needle Steering Using Reinforcement Learning Guided by Human Input

Full Article

Abstract

Citation

Autonomous Blood Suction for Robot-Assisted Surgery: A Sim-to-Real Reinforcement Learning Approach

Full Article

Abstract

Citation

A Realistic Surgical Simulator for Non-Rigid and Contact-Rich Manipulation in Surgeries with the da Vinci Research Kit

Full Article

Abstract

Code

CRESSim (Unity Project)

PhysX 5 for Unity

PhysX 5 for Unity Native Plugin (C++)

Citation

Towards Safe and Efficient Reinforcement Learning for Surgical Robots Using Real-Time Human Supervision and Demonstration

Full Article

Abstract

Citation

Robot Learning Incorporating Human Interventions in the Real World for Autonomous Surgical Endoscopic Camera Control

Full Article

Abstract

Citation

Sim-to-Real Surgical Robot Learning and Autonomous Planning for Internal Tissue Points Manipulation Using Reinforcement Learning

Full Article

Abstract

Citation

Site Changelog

2026-08-15

2026-01-22

When Everything Becomes `pip install`

当一切都变成 `pip install`