UROP Project

Vision-Language-Action Models for Robotic Assembly in Modular Construction

Vision-Language-Action Models (VLA); Construction Robotics; Modular Construction; Computer Vision; Robotic Assembly

Research Mentor: Prof. Juyeong Choi,
Department, College, Affiliation: Civil and Environmental Engineering, FAMU-FSU College of Engineering
Contact Email: jchoi3@fsu.edu
Research Assistant Supervisor (if different from mentor): Dr. Roshan Panahi Roshan
Research Assistant Supervisor Email: rp25q@fsu.edu
Faculty Collaborators:
Faculty Collaborators Email:

Looking for Research Assistants: Yes
Number of Research Assistants: 3
Relevant Majors: Relevant Research Assistant Major(s):

Computer Science
Electrical Engineering
Computer Engineering
Mechanical Engineering
Civil Engineering
Industrial Engineering
Project Location: RIDER Center (1753 W Paul Dirac Dr, Tallahassee, FL 32310)
Research Assistant Transportation Required: Yes
Remote or In-person: In-person
Approximate Weekly Hours: 7, During business hours
Roundtable Times and Zoom Link:

• Day: Monday, August 31
  Start Time: 3:30
  End Time: 4:00
  Zoom Link: https://fsu.zoom.us/j/99874344936?from=addon
• Day: Tuesday, September 1
  Start Time: 4:30
  End Time: 5:00
  Zoom Link: https://fsu.zoom.us/j/91434855422?from=addon
• Day: Wednesday, September 2
  Start Time: 3:30
  End Time: 4:00
  Zoom Link: https://fsu.zoom.us/j/98206142929?from=addon
• Day: Thursday, September 3
  Start Time: 4:30
  End Time: 5:00
  Zoom Link: https://fsu.zoom.us/j/97177409136?from=addon

Project Description

This project investigates the use of vision-language models (VLMs) and vision-language-action (VLA) models to enable more flexible robotic assembly in modular construction. Modular construction can improve productivity, quality, and construction speed, but customized production often requires frequent changes to manufacturing processes and robot programming. To address this challenge, the research will explore how artificial intelligence can allow a robotic arm to interpret visual information, understand natural-language instructions, and perform construction-related pick-and-place and assembly tasks with minimal task-specific programming. The project will integrate computer vision, Python programming, and robotic control to detect construction materials, determine their locations, and guide a robotic arm to manipulate them. The long-term goal is to support more adaptable and automated manufacturing systems for mass-customized modular construction.

Research Tasks: • Use ChatGPT, Claude, or a similar AI tool to: (1) generate code that instructs the robotic arm to pick up an object and place it at a specified location; (2) generate Python-based computer vision code to detect a piece of lumber in a video; and (3) detect the center point of the lumber and control the robotic arm to pick up the object using the detected coordinates.
• Learn about vision-language-action (VLA) models through online videos and other educational resources.
• Test the developed models in the lab using the robotic arm and provided software, and assist with data annotation, writing, and presentations as needed.


Skills that research assistant(s) may need: Requirements: Familiarity with Python programming. Prior experience with robotics projects is highly encouraged. Strong interest in robotics and artificial intelligence (AI). Self-motivated, disciplined, and willing to learn independently. Ability to attend in-person lab sessions for approximately 7 hours per week.