UROP Project

Vision-Language-Action Models for Robotic Assembly in Modular Construction

Modular Construction; Robotics; Vision-Language-Action Models

Research Mentor: Roshan Panahi, Dr.
Department, College, Affiliation: Civil and Environmental Engineering, FAMU-FSU College of Engineering
Contact Email: rpanahi@eng.famu.fsu.edu
Research Assistant Supervisor (if different from mentor):
Research Assistant Supervisor Email:
Faculty Collaborators:
Faculty Collaborators Email:

Looking for Research Assistants: Yes
Number of Research Assistants: 1
Relevant Majors: Computer Science; Electrical Engineering; Mechanical Engineering
Project Location: Walter Smith Architecture Building, 1938 S. Martin Luther King Jr. Blvd., Tallahassee, FL 32307-4200
Research Assistant Transportation Required: Yes
Remote or In-person: In-person
Approximate Weekly Hours: 7, Flexible schedule (Combination of business and outside of business. TBD between student and research mentor.)
Roundtable Times and Zoom Link:

• Day: Monday, August 31
  Start Time: 2:00
  End Time: 2:30
  Zoom Link: https://fsu.zoom.us/j/99225339098
• Day: Tuesday, September 1
  Start Time: 1:00
  End Time: 1:30
  Zoom Link: https://fsu.zoom.us/j/93987059441

Project Description

What if you could order a home customized to your preferences and have it manufactured and delivered to your site? This is the vision behind mass customization in modular construction. Modular construction is a construction method in which building components or modules are manufactured in a controlled factory environment and then transported to the project site for installation. Although this approach offers significant benefits in productivity, quality, and construction speed, producing highly customized homes in a factory remains challenging. Changes in design may require adjustments to production processes, tooling, workflows, and worker tasks. The workforce must also quickly understand and adapt to new designs, which can reduce productivity and increase the risk of errors. Robotics can help address some of these challenges by assisting workers with repetitive, labor-intensive, or frequently changing construction tasks. However, conventional industrial robots typically require task-specific programming, making frequent reprogramming for customized production costly and time-consuming. Recent advances in vision-language models (VLMs) and vision-language-action (VLA) models provide new opportunities for more flexible robotic systems. These models can enable robots to interpret visual information, understand natural-language instructions, and translate those instructions into actions. Therefore, this research aims to investigate the use of VLMs and VLA models to enable a robotic arm to perform construction-related pick-and-place and assembly tasks with minimal task-specific programming.

Research Tasks: (a) Use ChatGPT, Claude, or a similar AI tool to: (1) generate code that instructs the robotic arm to pick up an object and place it at a specified location; (2) generate Python-based computer vision code to detect a piece of lumber in a video; and (3) detect the center point of the lumber and control the robotic arm to pick up the object using the detected coordinates.
(b) Learn about vision-language-action (VLA) models through online videos and other educational resources.
(c) Test the developed models in the lab using the robotic arm and provided software, and assist with data annotation, writing, and presentations as needed.

Skills that research assistant(s) may need: Familiarity with Python programming. Prior experience with robotics projects is highly encouraged. Strong interest in robotics and artificial intelligence (AI). Self-motivated, disciplined, and willing to learn independently. Ability to attend in-person lab sessions for approximately 5–10 hours per week.