Dataset Access Integration Analysis Integration

Confidential AI Network Documentation

# Dataset Access Integration Analysis for Confidential AI Network

## Executive Summary

This document provides a comprehensive analysis of integrating dataset access capabilities with the Confidential AI Network application. The integration will enable secure, controlled, and auditable access to training datasets while maintaining privacy, compliance, and contractual obligations.

## 1. Current State Analysis

### 1.1 Existing Infrastructure

    * Backend: Node.js/Express with PostgreSQL database

    • Authentication: Keycloak IAM integration
    • Security: Blockchain verification and encrypted storage
    • Multi-Cloud: Support for AWS, Azure, GCP, OCI
    • Smart Contracts: Automated environment provisioning
    • ### 1.2 Current Dataset Management

    • Dataset metadata storage in PostgreSQL
    • Basic catalog functionality
    • Contract-based access control
    • Multi-party support framework

    ## 2. Integration Architecture

    ### 2.1 High-Level Architecture

    ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
    │   Frontend UI   │    │   API Gateway   │    │  Dataset Store  │
    │                 │    │                 │    │                 │
    │ - Dataset       │◄──►│ - Authentication │◄──►│ - S3/Azure Blob │
    │   Browser       │    │ - Authorization │    │ - GCP Storage   │
    │ - Upload Tools  │    │ - Rate Limiting │    │ - Local Storage │
    └─────────────────┘    └─────────────────┘    └─────────────────┘
             │                       │                       │
             │                       │                       │
             ▼                       ▼                       ▼
    ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
    │  Smart Contract │    │   Audit Log     │    │  Access Control │
    │   Verification  │    │   System        │    │   Engine        │
    │                 │    │                 │    │                 │
    │ - Contract      │    │ - Access Events │    │ - Role-Based    │
    │   Validation    │    │ - Usage Metrics │    │   Access        │
    │ - Payment       │    │ - Compliance    │    │ - Time-Based    │
    │   Processing    │    │   Reports       │    │   Restrictions  │
    └─────────────────┘    └─────────────────┘    └─────────────────┘

    ### 2.2 Core Components

    #### 2.2.1 Dataset Access Service
    // Core dataset access service
    class DatasetAccessService {
      async requestAccess(userId, datasetId, contractId) {
        // Validate contract terms
        // Check user permissions
        // Generate temporary access credentials
        // Log access request
      }
      
      async getDataset(userId, datasetId, accessToken) {
        // Verify access token
        // Apply data transformations
        // Return dataset with usage tracking
      }
      
      async revokeAccess(userId, datasetId) {
        // Invalidate access tokens
        // Update audit log
        // Notify relevant parties
      }
    }
    

    #### 2.2.2 Access Control Engine

    // Access control implementation
    class AccessControlEngine {
      async validateAccess(userId, datasetId, operation) {
        const contract = await this.getActiveContract(userId, datasetId);
        const userRole = await this.getUserRole(userId);
        
        return {
          allowed: this.checkPermissions(contract, userRole, operation),
          restrictions: this.getRestrictions(contract),
          auditRequired: this.isAuditRequired(operation)
        };
      }
    }
    

    ## 3. Technical Implementation

    ### 3.1 Database Schema Extensions

    #### 3.1.1 Dataset Access Tables
    -- Dataset access tracking
    CREATE TABLE dataset_access_logs (
      id SERIAL PRIMARY KEY,
      user_id INTEGER REFERENCES users(id),
      dataset_id INTEGER REFERENCES datasets(id),
      contract_id INTEGER REFERENCES contracts(id),
      access_type VARCHAR(50), -- 'read', 'download', 'stream'
      access_timestamp TIMESTAMP DEFAULT NOW(),
      ip_address INET,
      user_agent TEXT,
      success BOOLEAN,
      error_message TEXT
    );

    -- Temporary access tokens CREATE TABLE dataset_access_tokens ( id SERIAL PRIMARY KEY, user_id INTEGER REFERENCES users(id), dataset_id INTEGER REFERENCES datasets(id), token_hash VARCHAR(255) UNIQUE, expires_at TIMESTAMP, permissions JSONB, created_at TIMESTAMP DEFAULT NOW() );

    -- Dataset usage metrics CREATE TABLE dataset_usage_metrics ( id SERIAL PRIMARY KEY, dataset_id INTEGER REFERENCES datasets(id), user_id INTEGER REFERENCES users(id), bytes_accessed BIGINT, records_accessed INTEGER, access_duration INTERVAL, access_date DATE, created_at TIMESTAMP DEFAULT NOW() );

    ### 3.2 API Endpoints

    #### 3.2.1 Dataset Access APIs
    // RESTful API endpoints
    app.get('/api/datasets/:id/access', authenticate, async (req, res) => {
      const { id } = req.params;
      const userId = req.user.id;
      
      try {
        const access = await datasetAccessService.requestAccess(userId, id);
        res.json(access);
      } catch (error) {
        res.status(403).json({ error: error.message });
      }
    });

    app.get('/api/datasets/:id/download', authenticate, async (req, res) => { const { id } = req.params; const { accessToken } = req.query; try { const stream = await datasetAccessService.getDatasetStream(id, accessToken); res.setHeader('Content-Type', 'application/octet-stream'); stream.pipe(res); } catch (error) { res.status(403).json({ error: error.message }); } });

    app.post('/api/datasets/:id/revoke', authenticate, async (req, res) => { const { id } = req.params; const userId = req.user.id; try { await datasetAccessService.revokeAccess(userId, id); res.json({ success: true }); } catch (error) { res.status(500).json({ error: error.message }); } });

    ### 3.3 Frontend Integration

    #### 3.3.1 Dataset Browser Component

    // React component for dataset browsing
    const DatasetBrowser = () => {
      const [datasets, setDatasets] = useState([]);
      const [selectedDataset, setSelectedDataset] = useState(null);
      const [accessStatus, setAccessStatus] = useState({});

    const requestAccess = async (datasetId) => { try { const response = await api.post(/datasets/${datasetId}/access); setAccessStatus(prev => ({ ...prev, [datasetId]: 'granted' })); } catch (error) { setAccessStatus(prev => ({ ...prev, [datasetId]: 'denied' })); } };

    return (

    {datasets.map(dataset => ( requestAccess(dataset.id)} accessStatus={accessStatus[dataset.id]} /> ))}