Skip to main content

Storage

Configure cloud storage providers and perform file operations across AWS S3, Google Cloud Storage, and Azure Blob Storage through a unified interface.

Quick Example​

Use the storage API on the Ductape instance. Initialize Ductape with your access key:

import Ductape from '@ductape/sdk';

const ductape = new Ductape({
  accessKey: 'your-access-key',
});

// Upload a file
const result = await ductape.storage.upload({
  storage: 'main-storage',
  fileName: 'documents/report.pdf',
  buffer: fileBuffer,
  mimeType: 'application/pdf',
});

console.log('File URL:', result.url);

Supported Providers​

ProviderEnum ValueDescription
Amazon S3StorageProviders.AWSAWS S3 buckets
Google CloudStorageProviders.GCPGoogle Cloud Storage
AzureStorageProviders.AZUREAzure Blob Storage

File Operations​

Upload a File​

const result = await ductape.storage.upload({
  storage: 'main-storage',
  fileName: 'images/photo.jpg',
  buffer: imageBuffer,           // Buffer or string
  mimeType: 'image/jpeg',        // Optional
});

// Returns { success: true, url: '...', fileName: '...', mimeType: '...' }

Download a File​

const result = await ductape.storage.download({
  storage: 'main-storage',
  fileName: 'documents/report.pdf',
});

// Returns { success: true, data: Buffer, fileName?: string, size?: number, mimeType?: string }

Remove a File​

Use storage.remove() to delete a file:

const result = await ductape.storage.remove({
  storage: 'main-storage',
  fileName: 'documents/old-report.pdf',
});

// Returns { success: true, fileName?: string }

List Files​

Use storage.listFiles() for paginated listing:

const result = await ductape.storage.listFiles({
  storage: 'main-storage',
  prefix: 'documents/',           // Optional
  limit: 100,                     // Optional (default: 100, max: 1000)
  continuationToken: '...',       // Optional: for next page
});

// Returns { success: true, files: [...], limit?, nextToken?, hasMore }

Pagination Example​

Use continuationToken and nextToken for cursor-based pagination:

async function getAllFiles(product: string, env: string, storageTag: string) {
  const allFiles = [];
  let continuationToken: string | undefined;

  do {
    const result = await ductape.storage.listFiles({
      product,
      env,
      storage: storageTag,
      limit: 100,
      continuationToken,
    });

    allFiles.push(...result.files);
    continuationToken = result.nextToken;
  } while (continuationToken);

  return allFiles;
}

Get Storage Statistics​

Get file counts and sizes without loading all files:

const stats = await ductape.storage.stats({
  storage: 'main-storage',
  prefix: 'documents/',     // Optional
});

// Returns { success: true, totalFiles, totalSize, byType: { image, video, audio, document, archive, other } }

Generate Signed URL​

Create temporary URLs for secure file access:

// Read access (download)
const result = await ductape.storage.getSignedUrl({
  storage: 'main-storage',
  fileName: 'documents/report.pdf',
  expiresIn: 3600,            // Seconds (default: 3600)
  action: 'read',             // 'read' or 'write'
});

// Write access (upload)
const uploadUrl = await ductape.storage.getSignedUrl({
  storage: 'main-storage',
  fileName: 'uploads/new-file.pdf',
  expiresIn: 600,
  action: 'write',
});

Dispatch Storage Jobs​

Queue storage operations as background jobs:

const result = await ductape.storage.dispatch({
  storage: 'main-storage',
  operation: 'upload',
  input: {
    fileName: 'reports/monthly.pdf',
    buffer: reportBuffer,
    mimeType: 'application/pdf',
  },
  schedule: { start_at: Date.now() + 60000 },  // Optional: delay 1 minute
});

// Returns { job_id, status, scheduled_at, recurring?, next_run_at? }

Managing Storage Configurations​

Create, list, fetch, update, and delete storage configs with the CRUD API. See Storage CRUD for full examples.

// Create (cloud-linked)
await ductape.storage.create({ product: 'my-product', name: 'App Storage', tag: 'app-storage', envs: [...] });

// List / fetch / update / delete
await ductape.storage.list('my-product');
await ductape.storage.fetch('my-product', 'app-storage');
await ductape.storage.update('my-product', 'app-storage', { envs: [...] });
await ductape.storage.delete('my-product', 'app-storage');

Creating Storage Configurations​

Create storage with manual credentials or link a workspace cloud account by tag.

For cloud-linked storage, check available storage classes first:

const tiers = await ductape.cloud.tiers.list({ provider: 'aws', resource_type: 'storage' });
// tiers[0].tiers → [{ name: 's3-standard', label: 'Standard', est_cost_per_month: 0.023 }, …]
import { StorageProviders } from '@ductape/sdk/types';

// Cloud-linked (recommended)
await ductape.storage.create({
  product: 'my-product',
  name: 'App Storage',
  tag: 'app-storage',
  envs: [{
    slug: 'prd',
    type: StorageProviders.AWS,
    config: {
      cloud: 'prod_aws',
      bucketName: 'my-prod-bucket',
      region: 'us-east-1',
      tier: 's3-standard',   // from cloud.tiers.list() — controls storage class
    },
  }],
});

// Manual credentials
await ductape.storage.create({
  name: 'App Storage',
  tag: 'app-storage',
  envs: [
    {
      slug: 'prd',
      type: StorageProviders.AWS,
      config: {
        bucketName: 'my-prod-bucket',
        region: 'us-east-1',
        accessKeyId: process.env.AWS_ACCESS_KEY,
        secretAccessKey: process.env.AWS_SECRET_KEY,
      },
    },
  ],
});

See Cloud-linked components for the full AWS / GCP / Azure provisioning matrix, tier reference, and update flows.

Updating Storage​

await ductape.storage.update('my-product', 'app-storage', {
  envs: [{
    slug: 'prd',
    type: StorageProviders.GCP,
    config: {
      cloud: 'gcp_prod',
      bucketName: 'my-dev-bucket',
      location: 'US',
    },
  }],
});

Listing and Fetching Storage​

// List all storage configs for a product
const providers = await ductape.storage.list('my-product');

// Fetch a single storage by tag
const provider = await ductape.storage.fetch('my-product', 'app-storage');

API Reference​

ductape.storage Methods​

MethodDescription
create(data)Create a storage config (data includes product, name, tag, envs)
list(product)List all storage configs for a product
fetch(product, tag)Fetch a storage config by tag
update(product, tag, data)Update a storage config
delete(product, tag)Delete a storage config
upload(options)Upload a file
download(options)Download a file
remove(options)Remove (delete) a file
listFiles(options)List files with pagination
getSignedUrl(options)Generate a temporary signed URL
stats(options)Get file counts and sizes by type
dispatch(options)Queue a storage operation as a job
testConnection(options)Test storage provider connectivity

For reading a file from local disk (e.g. before upload), use ductape.storage.files.read(path).

Error Handling​

import { StorageError } from '@ductape/sdk';

try {
  const result = await ductape.storage.upload({ ... });
} catch (error) {
  if (error instanceof StorageError) {
    console.log('Error code:', error.code);
    console.log('Message:', error.message);
  }
}

Provider Configuration​

AWS S3​

{
  slug: 'prd',
  type: StorageProviders.AWS,
  config: {
    bucketName: 'your-bucket-name',
    accessKeyId: 'your-access-key-id',
    secretAccessKey: 'your-secret-access-key',
    region: 'us-east-1'
  }
}
FieldDescription
bucketNameThe name of the S3 bucket
accessKeyIdAWS access key ID
secretAccessKeyAWS secret access key
regionAWS region where the bucket is hosted

Google Cloud Storage​

{
  slug: 'prd',
  type: StorageProviders.GCP,
  config: {
    bucketName: 'your-gcp-bucket',
    config: {
      type: 'service_account',
      project_id: 'your-project-id',
      private_key_id: 'key-id',
      private_key: '-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n',
      client_email: 'service-account@project.iam.gserviceaccount.com',
      client_id: '123456789',
      auth_uri: 'https://accounts.google.com/o/oauth2/auth',
      token_uri: 'https://oauth2.googleapis.com/token',
      auth_provider_x509_cert_url: 'https://www.googleapis.com/oauth2/v1/certs',
      client_x509_cert_url: 'https://www.googleapis.com/robot/v1/metadata/x509/...',
      universe_domain: 'googleapis.com',
    },
  },
}
FieldDescription
bucketNameThe name of the GCP storage bucket
configService account credentials object (from GCP Console JSON key file)

Azure Blob Storage​

{
  slug: 'prd',
  type: StorageProviders.AZURE,
  config: {
    containerName: 'your-container-name',
    connectionString: 'your-connection-string',
  },
}
FieldDescription
containerNameThe name of the Azure Blob container
connectionStringThe connection string to access the container

Key Points​

  • Configure different providers per environment
  • Swap providers without code changes
  • Consistent interface across all providers
  • Built-in logging for all operations
  • Support for signed URLs with configurable expiry
  • Store credentials securely using environment variables

See Also​