Skip to main content

S3 Data Organizer

We migrated and extended the open-source Data Organizer, a frontend for managing S3-compatible storage solutions. As part of this work, we moved the application from Nuxt 2 to Nuxt 3, introduced a secure STS credentials flow, and modernized the testing and CI/CD processes.

Client/Company/Industry

GFZ Helmholtz Centre for Geosciences

Duration

7 months

Product

Open-Source Software

Expertise

Software Development

Goal

The goal of the project was to further develop the open-source frontend Data Organizer for managing S3-compatible storage solutions. To achieve this, the existing application was to be migrated from Nuxt 2 to Nuxt 3, extended with new features, and complemented by a modern testing and CI/CD infrastructure.

Tasks

  • Maintaining and managing tickets
  • Designing the single-page application architecture
  • Migrating the application from Nuxt 2 to Nuxt 3
  • Setting up and developing unit, Nuxt, and UI component tests with Vitest
  • Building and implementing end-to-end tests with Playwright
  • Setting up linting, type-checking, and formatting workflows
  • Further developing and optimizing the GitLab CI/CD pipelines
  • Containerizing the application with Docker
  • Revising and updating the project documentation
  • Extending the existing S3 mocking setup for test environments
  • Implementing the STS credentials flow for secure access to S3 resources
  • Developing new features with Vuetify
  • Implementing user experience improvements

Challenges

A key challenge was migrating the existing application from Nuxt 2 to Nuxt 3 without affecting the existing functionality. In addition, the STS credentials flow for secure access to S3 resources introduced additional complexity in both the authentication layer and the test environments.

Programming Languages

JavaScript/TypeScript, Python

Technologies

AWS SDK, Docker, GitLab CI/CD, Nuxt, OAuth2/OpenID Connect, Playwright, S3, STS, Vitest, Vue 3, Vuetify

Project Image

User interface of the Data Organizer showing an S3 bucket view, file management, and authentication status.

Similar problem?

Contact us

Takeaway

After the migration, the Data Organizer ran reliably on Nuxt 3 and had significantly expanded test coverage. The modernized architecture and the improved CI/CD pipeline created a solid foundation for further open-source development in the context of S3-based data platforms.

Similar Projects

Project Image

RIM2D - Highly Efficient 2D Hydraulic Simulation of Fluvial, Pluvial, and Urban Flooding

Hydrodynamic Simulation Web Application Geodata GPU Computing

RIM2D is an existing, highly efficient 2D hydraulic simulation model for fluvial, pluvial, and urban flooding. As part of a strategic partnership, we supported the extension of the research code with a web application and a cloud-based GPU simulation environment, enabling its transition into a market-ready product.

See project
Project Image

Data Lake for Geoscience Research Data Management

Datalake Research Data Management Geodata Cloud-Native Open-Source

We developed an Open-Source S3-based data lake solution for the centralized ingestion, categorization, and searchability of data. The goal was to automate and improve manual data management through an integrated architecture with workflow orchestration, data cataloging, and access control.

See project
All Projects
Back To Top