MEMON SYSTEMS

Architecting Zero-Allocation Telemetry System

Profile High-Frequency Telemetry Engine Blueprint (Global)
Stack Golang / Oracle Cloud (OCI) / Trino / Cloudflare
Duration 12 Weeks
Outcome 1M RPS (32 GB/s) at 98% Infrastructure OpEx Reduction

Video Walkthrough & Architecture Breakdown

Watch on YouTube ↗

The Challenge

At target ingest rate, off-the-shelf databases and hyperscaler managed services carried over $4,000,000/month in egress fees, alongside memory thrashing at the write path. Infrastructure cost scaled faster than the revenue the telemetry supported.

Executive Summary

This case study details the engineering journey of architecting a bespoke telemetry engine capable of sustaining 1,000,000 Requests Per Second (RPS) with a 32 KB payload per request (resulting in a continuous ingestion rate of 32 GB/s, or approximately 84 Petabytes (PB) of data per month). The architecture was required to deliver real-time 1:1 live reads, query capabilities for historical data by time range, and strict structural resilience, all while staying within a highly constrained operational expenditure (OpEx) ceiling of $175,000 monthly.

Traditional cloud-native designs and off-the-shelf database systems introduce catastrophic write amplification, excessive memory footprint, and massive egress overhead. By transitioning the infrastructure to Oracle Cloud Infrastructure (OCI), isolating compute and storage via cell-based configurations, and developing a custom, zero-allocation, memory-mapped (mmap) ring buffer engine in Go, we eliminated database compaction bottlenecks and network egress tax. This system achieved deterministic scale and static hardware utilization, driving monthly infrastructure expenditures below the target threshold.

1. The Core Mandate & Scale Economics

Operating a telemetry system for fully autonomous robots at a scale of 1,000,000 RPS introduces harsh physical constraints. Below is the baseline requirement and design math:

1.1 Sizing Calculations

To calculate the raw network throughput GG (in Gbps) required to ingest traffic given requests RR, payload size PP (in KB), packet and protocol overhead OO, and target hardware utilization UU:

G=R×P×8192×OU×109G = \frac{R \times P \times 8192 \times O}{U \times 10^9}

For our mandate:

  • Payload Size (PP): 32 KB=32,768 Bytes32\text{ KB} = 32,768\text{ Bytes}.
  • Ingestion Volume: 1,000,000 RPS1,000,000\text{ RPS}.
  • Sustained Throughput: 32 GB/s32\text{ GB/s} (256 Gbps256\text{ Gbps} raw network pipe).
  • Monthly Storage Volume: 32 GB/s×86,400 seconds/day×30.4 days/month≈84,050 TB≈84 PB32\text{ GB/s} \times 86,400\text{ seconds/day} \times 30.4\text{ days/month} \approx 84,050\text{ TB} \approx 84\text{ PB}.
  • Overhead (OO): 1.11.1 (due to protocol encapsulation over gRPC/TCP).
  • Target Infrastructure Utilization (UU): 70%70\% (to prevent queuing backpressure).

Using these inputs, the required global network capacity to sustain ingestion is:

G=1,000,000×32×8192×1.10.70×109≈411.94 GbpsG = \frac{1,000,000 \times 32 \times 8192 \times 1.1}{0.70 \times 10^9} \approx 411.94\text{ Gbps}

At this velocity, standard database layers collapse. The system requires concurrent read/write access:

  1. Live Read Path: Real-time 1:1 tapping of the telemetry stream for active dashboard monitoring.
  2. Historical Query Path: Index-driven access by arbitrary timestamp ranges to verify robotic state histories.
graph TD
%% ── Dark Elegant Theme ─────────────────────────────────
classDef crimson fill:#2a1111,stroke:#ff4444,stroke-width:1px,color:#ff8888,text-align:left
classDef prominent fill:#c8a96e,stroke:#c8a96e,color:#1a1508,font-weight:600
classDef ghost fill:#111110,stroke:#2a2926,stroke-width:1px,color:#6a6860,stroke-dasharray:4 3

    Goal["<b>MAIN REQUIREMENT</b><br>1,000,000 RPS Sustained | 32KB Payload"]:::prominent
    Math["<b>DESIGN MATH</b><br>32 Gigabytes per second (GB/s)<br>~84 Petabytes per month"]:::prominent
    Reads["<b>READ CAPABILITIES</b><br>1. Real-time 1:1 Live Read<br>2. Historical Data Access by Time Range"]:::prominent

    Goal ~~~ Math ~~~ Reads

2. Baseline Architecture & Systemic Failures

The industry standard approach to high-throughput ingestion relies on a relational engine backstopped by a fast caching layer. We analyzed and tested this baseline to verify its limits.

2.1 The PostgreSQL & Redis Buffer Collapse

In this traditional design, clients write to an API layer that puts data directly into an in-memory Redis cache cluster acting as a buffer. Background workers then poll Redis, bundle payloads, and write them sequentially to a persistent relational store (PostgreSQL).

flowchart LR
%% ── Dark Elegant Theme ─────────────────────────────────
classDef gold      fill:#1f1c15,stroke:#c8a96e,stroke-width:1.5px,color:#e8d5a8
classDef sage      fill:#131a14,stroke:#7a9e7e,stroke-width:1px,color:#a8c8ac
classDef steel     fill:#131620,stroke:#8896a8,stroke-width:1px,color:#b0bece
classDef crimson   fill:#1a1212,stroke:#a87878,stroke-width:1px,color:#d4a0a0
classDef ghost     fill:#111110,stroke:#2a2926,stroke-width:1px,color:#6a6860,stroke-dasharray:4 3
classDef prominent fill:#c8a96e,stroke:#c8a96e,color:#1a1508,font-weight:600

    Clients(("Client Applications")):::steel

    %% Critical Path at the Top
    subgraph HotPath ["Critical Path: Real-Time Ingestion"]
        direction LR
        NLB["Network Load Balancer"]:::gold
        App["Application Server"]:::prominent
        Redis[("Redis Cluster<br/>(High-Throughput Write Buffer)")]:::crimson
    end

    %% Background Path Below
    subgraph Background ["Asynchronous Persistence Tier"]
        direction LR
        Worker(("Batch Processor")):::ghost
        DB[("PostgreSQL<br/>(System of Record)")]:::sage
    end

    %% Flow connections
    Clients -- "Tx: Ingress Requests" --> NLB
    NLB -- "Rx: Egress Responses" --> Clients

    NLB -- "Tx: Routed Traffic" --> App
    App -- "Rx: Acknowledgements" --> NLB

    App -- "Tx: Low-Latency Memory Writes" --> Redis

    %% Background connections
    Redis -. "Rx: Asynchronous Polling" .-> Worker
    Worker -- "Tx: Bulk Persistence Operations" --> DB

Why it fails under load:

  1. Database Write Degradation: PostgreSQL forces the operating system to organize data pages on disk via locks and B-Trees. Under high-velocity writes, lock contention spikes, degrading disk write throughput to a fraction of the baseline.
  2. Drain Rate Discrepancy: The write velocity of the background workers into PostgreSQL is bounded by disk I/O, which is capped at ~50,000 IOPS on high-performance drives. Since ingestion writes occur at 1,000,000 RPS, the drain rate is slower than the ingestion rate.
  3. Buffer Saturated in 30 Seconds: At 32 GB/s, a 30-node Redis cluster with 1 TB of total RAM is filled to capacity in less than 32 seconds (1000 GB/32 GB/s=31.25 s1000\text{ GB} / 32\text{ GB/s} = 31.25\text{ s}). Once the memory buffer saturates, the API layer blocks incoming socket connections, causing immediate timeouts, packet drops, and catastrophic data loss.
  4. Runtime and Memory Overhead: Utilizing large nodes running NodeJS with PM2 or Python workers introduces massive garbage collection (GC) and Inter-Process Communication (IPC) overhead, consuming resources simply to maintain socket states.

2.2 The Hyperscaler Managed Services Cost Trap

To scale past local database bottlenecks, we evaluated cloud-native managed services on Amazon Web Services (AWS) using Auto-Scaling Groups, Amazon Managed Streaming for Apache Kafka (MSK), ElastiCache (Redis), and Amazon S3.

graph TD
    %% ── Classes ─────────────────────────────────────────────
    classDef gold      fill:#1f1c15,stroke:#c8a96e,stroke-width:1.5px,color:#e8d5a8
    classDef sage      fill:#131a14,stroke:#7a9e7e,stroke-width:1px,color:#a8c8ac
    classDef steel     fill:#131620,stroke:#8896a8,stroke-width:1px,color:#b0bece
    classDef crimson   fill:#1a1212,stroke:#a87878,stroke-width:1px,color:#d4a0a0
    classDef ghost     fill:#111110,stroke:#2a2926,stroke-width:1px,color:#6a6860,stroke-dasharray:4 3
    classDef prominent fill:#c8a96e,stroke:#c8a96e,color:#1a1508,font-weight:600

    %% 1. Input Source
    Users(["Clients"]):::prominent

    %% 2. Entry Point
    Users -->|Incoming Requests| LB["Load Balancer"]:::steel

    %% Region Boundary
    subgraph Region["Cloud Region"]
        direction TB

        %% Availability Zone 1
        subgraph AZ1["Availability Zone 1"]
            direction TB
            Compute_AZ1["Compute Instances"]:::steel
            MQ_AZ1["Message Queue"]:::sage
            Cache_AZ1["Distributed Cache"]:::crimson
            
            Compute_AZ1 -->|Write Events| MQ_AZ1
            Compute_AZ1 -->|Read/Write| Cache_AZ1
        end

        %% Availability Zone 2
        subgraph AZ2["Availability Zone 2"]
            direction TB
            Compute_AZ2["Compute Instances"]:::steel
            MQ_AZ2["Message Queue"]:::sage
            Cache_AZ2["Distributed Cache"]:::crimson
            
            Compute_AZ2 -->|Write Events| MQ_AZ2
            Compute_AZ2 -->|Read/Write| Cache_AZ2
        end

        %% 4. Data Layer Synchronization
        MQ_AZ1 -.->|Sync| MQ_AZ2
        Cache_AZ1 -.->|Sync| Cache_AZ2
    end

    %% 3. Scaling Mechanics (Moved outside Region for layout)
    subgraph Scaling["Auto-Scaling"]
        direction LR
        HPA["Horizontal Scaling"]:::ghost
        CA["Node Scaling"]:::ghost
    end

    LB -->|Distributes Traffic| Compute_AZ1
    LB -->|Distributes Traffic| Compute_AZ2

    %% Scaling Interactions
    Scaling -.->|Monitors and Scales| Compute_AZ1
    Scaling -.->|Monitors and Scales| Compute_AZ2

While cloud-native managed services resolve write limits through automatic scaling, the utility cost at this ingest rate is the constraint:

graph TD
    %% ── Classes ─────────────────────────────────────────────
    classDef gold      fill:#1f1c15,stroke:#c8a96e,stroke-width:1.5px,color:#e8d5a8
    classDef sage      fill:#131a14,stroke:#7a9e7e,stroke-width:1px,color:#a8c8ac
    classDef steel     fill:#131620,stroke:#8896a8,stroke-width:1px,color:#b0bece
    classDef crimson   fill:#1a1212,stroke:#a87878,stroke-width:1px,color:#d4a0a0
    classDef ghost     fill:#111110,stroke:#2a2926,stroke-width:1px,color:#6a6860,stroke-dasharray:4 3
    classDef prominent fill:#c8a96e,stroke:#c8a96e,color:#1a1508,font-weight:600

    %% ── Main Header ─────────────────────────────────────────
    TotalCost(["Monthly Baseline: ~$8,642,000"]):::prominent

    %% ── List Layout Using Subgraphs & Links ─────────────────

    subgraph Network ["1. Network ($5.76M)"]
        direction TB
        Egress["Internet Egress (84 PB @ $0.05/GB)<br/>$4,200,000"]:::crimson
        InterAZ["Inter-AZ Transfer (66% @ $0.02/GB)<br/>$1,120,000"]:::crimson
        OtherNetwork["..."]:::ghost
        
        Egress --- InterAZ --- OtherNetwork
    end

    subgraph Compute ["2. Load Balancing & Compute ($990k)"]
        direction TB
        LB["Load Balancer (~32k LCU)<br/>$140,000"]:::steel
        OtherCompute["..."]:::ghost
        
        LB --- OtherCompute
    end

    subgraph Storage ["3. Storage ($1.9M+)"]
        direction TB
        S3M1["Object Storage Month 1 (84 PB @ $0.021/GB)<br/>$1,764,000"]:::crimson
        S3M23["Object Storage Month 2 (168 PB): $3,528,000<br/>Object Storage Month 3 (252 PB): $5,292,000"]:::crimson
        OtherStorage["..."]:::ghost
        
        S3M1 --- S3M23 --- OtherStorage
    end

    subgraph Traps ["4. Fatal Cost Traps"]
        direction TB
        NAT["Ingress / NAT Gateway Fee (84 PB)<br/>+$3,780,000 / mo"]:::ghost
        S3Unbatched["Unbatched Storage Puts (2.6T Requests)<br/>+$12,900,000 / mo"]:::ghost
        LogAll["Unsampled Logging<br/>Unlimited Hidden Cost"]:::ghost
        
        NAT --- S3Unbatched --- LogAll
    end

    %% Connections from Total
    TotalCost --> Network
    TotalCost --> Compute
    TotalCost --> Storage
    TotalCost --> Traps
  1. Egress Fees: To serve the live dashboard reads, approximately 84 PB of data must transit the cloud boundary. On AWS, egress rates of $0.05/GB\$0.05/\text{GB} generate a monthly network bill of $4,200,000.
  2. Inter-Availability Zone (AZ) Taxes: Replicating data across AZs for high availability costs $0.02/GB\$0.02/\text{GB} (billed in both directions) for inter-AZ data transfer. Assuming 66% of the 32 GB/s traffic traverses AZ boundaries, this adds $1,120,000 monthly.
  3. Storage Ingestion Fees (PUT APIs): Sending unbatched 32 KB telemetry payloads to S3 generates 2.59 trillion API calls per month. At $0.005\$0.005 per 1,000 PUT requests, this incurs an additional $12,950,000 monthly storage transaction fee.
  4. Compounding Storage Baseline: Standard S3 storage costs $0.021/GB\$0.021/\text{GB}. Storing 84 PB in Month 1 costs $1,764,000, which compounds to $3,528,000 in Month 2, and $5,292,000 in Month 3.

3. The Architectural Iterations & Progression

To navigate these limits, we executed four distinct architectural iterations, refining the physical and financial performance at each stage.

3.1 Iteration 1: Cloud-Native Managed Services (AWS Optimization)

We attempted to optimize the AWS topology by reducing the payload size. However, we realized that the 32 KB payload was already a pre-compressed telemetry block (e.g., packed binary format or compressed JSON), meaning further compression was physically impossible. We had to ingest and persist the full 32 KB payload at 1,000,000 RPS, keeping the network demand at a relentless 256 Gbps (32 GB/s) baseline. We utilized Route 53 with latency-based routing to 10x Network Load Balancers and ran local Redis caches synced eventually to ScyllaDB on AWS EC2 nodes.

  • Result: The inability to compress the payload meant network requirements remained at 256 Gbps. Coupled with NAT gateway processing fees ($0.045 per GB) and inter-AZ network charges, the monthly OpEx baseline stayed over $1,200,000, rendering this optimization attempt a failure.

3.2 Iteration 2: Kernel-Bypassed Seastar & OCI Hybrid Stack

To eliminate AWS’s high network fees, we moved the entire infrastructure to Oracle Cloud Infrastructure (OCI), which offers free inbound traffic, low egress rates, and unmetered internal VCN transit.

To bypass kernel context switches, we implemented a C++ Seastar-based stack: a Golang API fleet writing concurrently to a Redpanda queue broker (built on the Seastar thread-per-core framework) for fast, append-only logs, and to a Dragonfly cache cluster for high-velocity reads. A background worker fleet drained Redpanda into a ScyllaDB ring deployed on local NVMe drives.

graph TD
classDef global fill:#1a365d,stroke:#2b6cb0,stroke-width:2px,color:#fff;
classDef compute fill:#2d3748,stroke:#4a5568,stroke-width:2px,color:#fff;
classDef fast fill:#d69e2e,stroke:#b7791f,stroke-width:2px,color:#fff;
classDef slow fill:#2c7a7b,stroke:#38b2ac,stroke-width:2px,color:#fff;

Client((Global Users))

subgraph The_Edge_Layer [Edge Layer]
    CF_DNS[Cloudflare DNS \n Traffic Routing & DDoS Protection]:::global
    CF_CDN[Cloudflare CDN \n Static Cache & Edge Routing]:::global
end

subgraph LB_Layer [Load Balancing]
    HAProxy[OCI NLB \n Network Load Balancer]:::fast
end

subgraph VM_BareMetal_Infrastructure [Cloud Infrastructure]
    subgraph Compute_Layer [Compute Fleet]
        API[Golang API Fleet \n Fiber on ARM]:::compute
    end

    subgraph Fast_Path [The Fast Path - Milliseconds]
        Dragonfly[(Dragonfly Cluster \n Multi-threaded Cache)]:::fast
        Redpanda[Redpanda Brokers \n Append-Only Write Log]:::fast
    end

    subgraph Async_Slow_Path [The Async/Slow Path]
        Worker[Go Background Workers \n Batch Processing]:::compute
        Scylla[(ScyllaDB Ring \n Permanent Storage)]:::slow
        S3[(OCI Object Storage / S3 \n Cold Data Archive)]:::slow
    end
 end

Client -->|1- DNS Request| CF_DNS
CF_DNS -.->|2- Resolved LB IP| Client
Client ==>|3- HTTP Request| CF_CDN
CF_CDN ==>|4- Cache Miss / Route to Origin| HAProxy
HAProxy ==>|5- L4 TCP Connection| API
API ==>|6a- Append Log| Redpanda
API ==>|6b- Cache Recent Write| Dragonfly
API -.->|6c- Fetch Recent Data| Dragonfly
Redpanda ==>|7- Batch Consume| Worker
Worker ==>|8a- Bulk Insert| Scylla
Worker -.->|8b- Evict Temp Key| Dragonfly
Worker -.->|8c- Soft Purge Surrogate Tag| CF_CDN
API -.->|9- Fetch Hot Data| Scylla
API -.->|9a- Go Singleflight Coalescing| API
API -.->|9b- Write Frequent History - Hot Data - to Cache| Dragonfly
Worker -.->|10a- Fetch Oldest/Least Read Data| Scylla
Worker -.->|10b- Offload & Archive| S3

The failure modes:

  1. Raft Consensus Multiplier: Redpanda requires a Raft consensus quorum. To guarantee high availability, replication factor (RF) must be set to at least 3. This tripled the required node count and disk capacity, driving compute costs beyond our budget.
  2. Local NVMe Storage Limit: ScyllaDB was bound to local NVMe drives. When a node reached its storage limit, the cluster required scaling.
  3. Data Loss Risk: Because data sat on local NVMe, if a node crashed, its unreplicated data block was lost, exposing the system to data loss.
  4. Network Bandwidth Saturation: Servicing concurrent writes, cache updates, and background consumer drains pushed the VMs' Virtual NICs (VNICs) to their limit, causing packet loss.

3.3 Iteration 3: Bare ScyllaDB & Network Disk Compaction Bottleneck

To simplify the topology, we removed Redpanda and Dragonfly, routing the Golang API fleet directly to a ScyllaDB cluster. To prevent data loss from node failures, we moved ScyllaDB from local NVMe to OCI Elastic Network Block Volumes, enabling automated snapshot backups.

graph TD

%% Define Classes

classDef global fill:#1a365d,stroke:#2b6cb0,stroke-width:2px,color:#fff;

classDef compute fill:#2d3748,stroke:#4a5568,stroke-width:2px,color:#fff;

classDef storage fill:#2c7a7b,stroke:#38b2ac,stroke-width:2px,color:#fff;

  

%% New High-Contrast class for Crucial Functionality & Testing

classDef crucial fill:#e53e3e,stroke:#c53030,stroke-width:4px,color:#fff,stroke-dasharray: 5 5;

  

Client((Global Users))

  

subgraph The_Edge_Layer [Edge Layer - Optional for Core Path]

CF_DNS[Cloudflare DNS \n Traffic Routing]:::global

CF_CDN[Cloudflare CDN \n Edge Security]:::global

end

  

%% CRUCIAL: High-throughput entry point must be tested

subgraph LB_Layer [LB Layer]

NLB[OCI NLB \n L4 Stateless Balancing]:::crucial

end

  

subgraph Cloud_Infrastructure [OCI Cell Infrastructure - 1 of 8]

%% CRUCIAL: API logic and Shard-Awareness is critical for performance

subgraph Compute_Layer [API Fleet]

API[Golang API Fleet \n 5x A1.Flex Nodes]:::crucial

end

  

%% CRUCIAL DB / Optional S3

subgraph Storage_Layer [Unified Data Layer]

Scylla[(ScyllaDB Cluster \n 1.5TB Block Volumes / 120 VPU)]:::crucial

ObjStorage[(OCI Object Storage \n Tiered Storage / Archive)]:::storage

end

end

  

Client -->|1| CF_DNS

CF_DNS -.->|2| Client

Client ==>|3| CF_CDN

CF_CDN ==>|4| NLB

NLB ==>|5| API

  

%% CRUCIAL: High-load data path

API ==>|6 - Critical Write/Read| Scylla

Scylla -.->|7| API

Scylla -.->|8 - Tiering| ObjStorage

The failure modes:

  1. LSM Tree Write Amplification: ScyllaDB utilizes a Log-Structured Merge (LSM) tree. This architecture writes to a Memtable, flushes to SSTables on disk, and runs compaction cycles. Compaction cycles multiply the raw write throughput by a Write Amplification Factor (WAF) of 4.
  2. Commit Log Contention: In addition to SSTables, ScyllaDB writes each incoming payload to a Commit Log on disk, adding 32 GB/s32\text{ GB/s} of overhead.
  3. Disk Performance Bottleneck: To support a WAF of 4 (128 GB/s128\text{ GB/s}) and the Commit Log (32 GB/s32\text{ GB/s}), the network disks required a sustained throughput of 160 GB/s160\text{ GB/s}. Pushing this volume of I/O over network blocks saturated OCI's network interface limits. This caused write queuing, latency spikes, and system failure, proving that LSM-based engines cannot be run directly over network block volumes at this scale.

3.4 Iteration 4: Aerospike Hybrid Storage & Apache Pulsar/Bookkeeper

We replaced ScyllaDB with Aerospike DB operating in hybrid mode (indexes in RAM, payloads on raw SSDs). To handle disaster recovery, we introduced Apache Pulsar with an Apache BookKeeper storage fleet. BookKeeper was configured with DbLedgerStorage (memory-mapped index) and double writes disabled (journalWriteData=false), storing data on network block volumes backed up by weekly snapshots.

To reduce Network Load Balancer (NLB) load, we implemented a Smart Client Routing connection flow.

graph TD
    %% ── Dark Elegant Theme ─────────────────────────────────
    classDef gold      fill:#1f1c15,stroke:#c8a96e,stroke-width:1.5px,color:#e8d5a8
    classDef sage      fill:#131a14,stroke:#7a9e7e,stroke-width:1px,color:#a8c8ac
    classDef steel     fill:#131620,stroke:#8896a8,stroke-width:1px,color:#b0bece
    classDef crimson   fill:#1a1212,stroke:#a87878,stroke-width:1px,color:#d4a0a0
    classDef ghost     fill:#111110,stroke:#2a2926,stroke-width:1px,color:#6a6860,stroke-dasharray:4 3
    classDef prominent fill:#c8a96e,stroke:#c8a96e,color:#1a1508,font-weight:600

    %% ── Subgraph Styles ───────────────────────────────────
    style Compute_Fleet fill:#181922,stroke:#8896a8,stroke-width:1.5px,stroke-dasharray:5 5
    style Specs_Compute fill:none,stroke:none
    
    style BK_Fleet fill:#1d1616,stroke:#a87878,stroke-width:1.5px,stroke-dasharray:5 5
    style Specs_BK fill:none,stroke:none

    style DB_Fleet fill:#161d17,stroke:#7a9e7e,stroke-width:1.5px,stroke-dasharray:5 5
    style Specs_DB fill:none,stroke:none

    NLB[OCI NLB]:::gold
    
    subgraph Compute_Fleet [Compute Fleet]
        subgraph Specs_Compute [Compute Specifications]
            Info_Compute["Total Nodes: 20<br>Shape: VM.Standard.A1.Flex (40 OCPU)<br>Workload: Go API + Apache Pulsar Brokers<br>Network: 800 Gbps Capacity"]:::prominent
        end
        
        VM1[Compute Node 1]:::steel
        VM2[Compute Node 2]:::steel
        VMDots[...]:::ghost
        VM20[Compute Node 20]:::steel
    end

    subgraph BK_Fleet [Apache BookKeeper Fleet]
        subgraph Specs_BK [BookKeeper Specifications]
            Info_BK["Total Nodes: 36<br>Shape: VM.Standard.A1.Flex (40 OCPU)<br>Config: E=2, WQ=2, AQ=1 (WAF 1, RF 2)"]:::prominent
        end
        
        BK1[Bookie Node 1]:::crimson
        BK2[Bookie Node 2]:::crimson
        
        BKDots[...]:::ghost
        BK36[Bookie Node 36]:::crimson
    end

    subgraph Storage_Fleet [Block Storage Fleet]
        subgraph Specs_Storage [Storage Specifications]
            Info_Storage["Standard Block Volumes<br>Base 36, Scales to 112 disks max<br>Backups: Weekly Snapshots"]:::prominent
        end
        
        Vol1[(Block Volume 1)]:::ghost
        Vol2[(Block Volume 2)]:::ghost
    end

    BK1 --- Vol1
    BK2 --- Vol2

    subgraph DB_Fleet [Aerospike Database Fleet]
        subgraph Specs_DB [DB Specifications]
            Info_DB["Total Nodes: 12<br>Shape: VM.DenseIO.E4.Flex<br>Configuration: RF 1 (per cell), Hybrid Mode"]:::prominent
        end
        
        DB1[Aerospike Node 1]:::sage
        DB2[Aerospike Node 2]:::sage
        
        DBDots[...]:::ghost
        DB12[Aerospike Node 12]:::sage
    end
    
    NLB -- "256 Gbps" --> VM1
    NLB --> VM2
    NLB -.-> VMDots
    NLB --> VM20

    Compute_Fleet -- "256 Gbps<br>Dual Write (Backup & Recovery)" --> BK_Fleet
    Compute_Fleet -- "256 Gbps<br>Dual Write (Main DB)" --> DB_Fleet

Smart Client Connection Sequence:

  1. Discovery Phase: The application client queries the OCI NLB on port 3000 as a seed node.
  2. Topology Discovery: The seed node returns a partition mapping containing the real private IPs of all Aerospike cluster nodes.
  3. Bypass NLB: The client establishes direct TCP connections to each Aerospike node IP. The NLB path is abandoned, directing subsequent data traffic straight to the target nodes.
sequenceDiagram
    autonumber
    participant Client App as Application (Smart Client)
    box "OCI Network Infrastructure" #fee
    participant NLB as OCI NLB L4 (Discovery Only)
    end
    box "Aerospike Cluster (Subnet B)" #e1f5fe
    participant NodeA as Aerospike Node A
    participant NodeB as Aerospike Node B (Owner of 'X')
    end

    Note over Client App, NodeA: PHASE 1: DISCOVERY (Low Bandwidth)
    Note right of Client App: App Startup:<br/>Points to NLB IP as 'Seed'
    Client App->>NLB: [SEED REQ] Where is the cluster?
    activate NLB
    NLB->>NodeA: Forward to one node
    deactivate NLB
    activate NodeA
    
    Note right of NodeA: node.conf:<br/>advertises REAL Private IPs<br/>(access-address)
    NodeA-->>NLB: Returns Partition Map & Node IPs
    activate NLB
    NLB-->>Client App: Returns Partition Map & Node IPs
    deactivate NLB
    deactivate NodeA
    
    Note over Client App: Smart Client opens DIRECT TCP connections<br/>to every Node IP. The NLB path is now abandoned.

    Note over Client App, NodeB: PHASE 2: HYPERSCALE DATA FLOW (32 GB/s)
    
    Note right of Client App: Req Record 'X'.<br/>Smart Map says "Node B owns 'X'"
    Client App->>NodeB: [DIRECT REQ] Send me Record 'X'
    activate NodeB
    NodeB-->>Client App: Return Record 'X'
    deactivate NodeB

    Note over NLB: NLB is IDLE.<br/>32 GB/s traffic flows directly<br/>over individual node NICs.

To limit network utilization, we configured BookKeeper’s skipListSizeLimit to 2 GB. This aggregated incoming 32 KB payloads in RAM and wrote them to disk sequentially in large chunks.

To reduce data egress charges, we routed historical queries to a flat-rate OCI FastConnect port (400 Gbps400\text{ Gbps} connection linked to collocated Arista routers), bypassing standard egress fees.

graph TD
    %% ── Dark Elegant Theme ─────────────────────────────────
    classDef gold      fill:#1f1c15,stroke:#c8a96e,stroke-width:1.5px,color:#e8d5a8
    classDef sage      fill:#131a14,stroke:#7a9e7e,stroke-width:1px,color:#a8c8ac
    classDef steel     fill:#131620,stroke:#8896a8,stroke-width:1px,color:#b0bece
    classDef crimson   fill:#1a1212,stroke:#a87878,stroke-width:1px,color:#d4a0a0
    classDef ghost     fill:#111110,stroke:#2a2926,stroke-width:1px,color:#6a6860,stroke-dasharray:4 3
    classDef prominent fill:#c8a96e,stroke:#c8a96e,color:#1a1508,font-weight:600
    classDef rejected  fill:#2a1111,stroke:#ff4444,stroke-width:2px,color:#ff8888,stroke-dasharray:5 5

    %% ── Subgraph Styles ───────────────────────────────────
    style Compute_Fleet fill:#181922,stroke:#8896a8,stroke-width:1.5px,stroke-dasharray:5 5
    style Specs_Compute fill:none,stroke:none
    
    style DB_Fleet fill:#161d17,stroke:#7a9e7e,stroke-width:1.5px,stroke-dasharray:5 5
    style Specs_DB fill:none,stroke:none
    
    style XDR_Storage fill:#1d1616,stroke:#ff4444,stroke-width:2px,stroke-dasharray:10 5
    style Specs_XDR fill:none,stroke:none

    NLB[OCI NLB]:::gold
    
    subgraph Compute_Fleet [Compute Fleet]
        subgraph Specs_Compute [Compute Specifications]
            Info_Compute["Total Nodes: 20<br>Shape: VM.Standard.A1.Flex (40 OCPU)<br>Workload: Go API<br>Network: 800 Gbps Capacity"]:::prominent
        end
        
        VM1[Compute Node 1]:::steel
        VM2[Compute Node 2]:::steel
        VMDots[...]:::ghost
        VM20[Compute Node 20]:::steel
    end

    subgraph DB_Fleet [Aerospike Database Fleet]
        subgraph Specs_DB [DB Specifications]
            Info_DB["Total Nodes: 12<br>Shape: VM.DenseIO.E4.Flex<br>Configuration: RF 2, Hybrid Mode"]:::prominent
        end
        
        DB1[Aerospike Node 1]:::sage
        DB2[Aerospike Node 2]:::sage
        
        DBDots[...]:::ghost
        DB12[Aerospike Node 12]:::sage
    end

    subgraph XDR_Storage ["XDR Recovery / Backup Storage (Rejected)"]
        subgraph Specs_XDR [XDR Specifications]
            Info_XDR["OCI Object Storage / Network Volume<br>Requires extra 256 Gbps reserved pipe<br>High Bandwidth Consumption & Latency"]:::rejected
        end
        
        OS1[(Object Storage / Vol)]:::rejected
    end
    
    NLB -- "256 Gbps" --> VM1
    NLB --> VM2
    NLB -.-> VMDots
    NLB --> VM20

    Compute_Fleet -- "256 Gbps<br>Write (Main DB)" --> DB_Fleet
    
    DB_Fleet == "XDR Sync<br>⚠️ Extra 256 Gbps Pipe Required" === XDR_Storage
    linkStyle 5 stroke:#ff4444,stroke-width:3px,color:#ff8888,stroke-dasharray:5 5

The configuration specifications:

1. aerospike.conf Block Engine Profile
# Aerospike Community Edition Configuration
service {
    user                 root
    group                root
    pidfile              /var/run/aerospike/asd.pid
    paxos-single-replica-limit  1      # Single replica mapping (RF=1) per cell
    service-threads      80            # 2x OCPU allocation for core pinning
    transaction-queues   80
    transaction-threads-per-queue  4
    proto-fd-max         200000
}

namespace test {
    replication-factor   1
    memory-size          60G           # DRAM allocated for hot indexes
    default-ttl          0             # Never expire writes

    # Direct NVMe interface bypasses OS Page Cache
    storage-engine device {
        device           /dev/sdb
        device           /dev/sdc
        device           /dev/sdd
        device           /dev/sde
        device           /dev/sdf
        device           /dev/sdg
        device           /dev/sdh
        device           /dev/sdi      # 8x iSCSI volumes

        write-block-size     1048576   # 1 MB blocks to optimize disk writes
        read-page-cache      false
        post-write-queue     256
        defrag-lwm-pct       50        # Defrag blocks at <50% occupancy
        defrag-sleep         100       # Wait 100us between defrag cycles
        max-write-cache      128M
    }
}
2. bookkeeper.conf Ingestion Settings
# Apache BookKeeper Settings for Ingestion
ledgerStorageClass=org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
journalWriteData=false
skipListSizeLimit=2147483648
diskUsageThreshold=0.95

The failure modes:

  1. Licensing Costs: Aerospike Enterprise licensing costs scale with storage volume. At a data volume of 84 PB per month, the licensing fees exceeded our budget limits.
  2. Cell Boundaries limits: While cell-based clusters can isolate storage nodes to stay within community licensing bounds, managing multi-datacenter data consistency added significant operational overhead.
  3. Write Failures during Network Slowdowns: If OCI network transit slowed down, Aerospike’s memory buffers filled up and rejected write operations, causing data loss.

4. The Final Architecture: State Machine Ring Buffer & Custom Storage Engine

To meet both physical and budget requirements, we designed a custom telemetry engine that replaces third-party database systems. The core design shifts data persistence from off-the-shelf databases to a custom, memory-mapped write-ahead log (WAL) that offloads data to OCI Object Storage.

graph TD
%% ── Dark Elegant Theme ─────────────────────────────────
classDef gold      fill:#1f1c15,stroke:#c8a96e,stroke-width:1.5px,color:#e8d5a8
classDef sage      fill:#131a14,stroke:#7a9e7e,stroke-width:1px,color:#a8c8ac
classDef steel     fill:#131620,stroke:#8896a8,stroke-width:1px,color:#b0bece
classDef crimson   fill:#1a1212,stroke:#a87878,stroke-width:1px,color:#d4a0a0
classDef ghost     fill:#111110,stroke:#2a2926,stroke-width:1px,color:#6a6860,stroke-dasharray:4 3
classDef prominent fill:#c8a96e,stroke:#c8a96e,color:#1a1508,font-weight:600

    subgraph OCI ["Oracle Cloud Infrastructure"]
        subgraph Core ["Core Ingestion Architecture"]
            NLB["Network Load Balancer (Layer 4)"]:::prominent
            VMPool["Compute Pool<br>(40 OCPU / 128GB RAM VMs)<br>Distributed across Fault Domains"]:::steel
            NetDisks[("Elastic Network Disks<br>(Buffer & WAL)")]:::gold
        end

        subgraph StorageTier ["Data & Analytics Tier"]
            ObjStore[("OCI Object Storage<br>(Persistent Archive)")]:::sage
            Trino["OCI Big Data Service<br>(Trino / Historical Query)"]:::gold
        end
    end

    NLB -- "1M RPS" --> VMPool
    VMPool -- "Appends to" --> NetDisks
    NetDisks -- "Offloads Frozen Data" --> ObjStore
    ObjStore -- "Historical Analytics" --> Trino
    class OCI ghost

4.1 Cell-Based Architecture

To stay within OCI network limits and manage risk, we split the system into 4 isolated cells.

  • Cell Capacity: Each cell is configured to handle 250,000 RPS (65 Gbps, 8 GB/s ingestion throughput).
  • Isolation Policy: Cells share no network pathing, load balancers, or metadata synchronization, preventing cascade failures.
  • Universal Scalability Law (USL) Optimization: The cell-based design keeps our node count within the linear scaling range of the USL curve. This prevents the performance degradation that occurs when large clusters spend more CPU time coordinating states than processing data.

4.2 Stateless Compute & Storage Layout

Each cell contains a stateless compute fleet and an elastic storage tier.

Compute Nodes (12x per Cell, 48x globally):

  • Shape: VM.Standard.A1.Flex (40 Ampere Altra ARM Cores, 128 GB RAM, 50 GB Boot Volume).
  • Network Capacity: OCI allocates network throughput based on CPU core count. A 40 OCPU instance receives 40 Gbps (5,000 MB/s) of full-duplex network bandwidth.
  • Base Utilization Target: 53.33% (2.68 GB/s2.68\text{ GB/s} of network capacity consumed per node during normal operation).

Hardware Resiliency Justification:

The compute nodes are distributed evenly across 3 OCI Fault Domains (FD) (4 VMs per FD).

  • Fault Tolerance: If a fault domain goes offline, we lose 4 VMs. The remaining 8 VMs must pick up the cell load of 250,000 RPS.
  • Node Load Margin: With 8 VMs running, the utilization per node rises to:

Utilization=12 VMs×53.33%8 VMs=80%\text{Utilization} = \frac{12 \text{ VMs} \times 53.33\%}{8 \text{ VMs}} = 80\%

This stays within the safe 80% operating limit, preventing system crashes. If we configured the cell with only 9 VMs (running at 71.1% base utilization), a fault domain failure (losing 3 VMs) would push the remaining 6 VMs to:

Utilization=9 VMs×71.11%6 VMs≈106.6%\text{Utilization} = \frac{9 \text{ VMs} \times 71.11\%}{6 \text{ VMs}} \approx 106.6\%

This exceeds VM capacity, causing immediate node crashes and cascade failure.

graph TD
%% ── Dark Elegant Theme ─────────────────────────────────
classDef gold      fill:#1f1c15,stroke:#c8a96e,stroke-width:1.5px,color:#e8d5a8
classDef sage      fill:#131a14,stroke:#7a9e7e,stroke-width:1px,color:#a8c8ac
classDef steel     fill:#131620,stroke:#8896a8,stroke-width:1px,color:#b0bece
classDef crimson   fill:#1a1212,stroke:#a87878,stroke-width:1px,color:#d4a0a0
classDef ghost     fill:#111110,stroke:#2a2926,stroke-width:1px,color:#6a6860,stroke-dasharray:4 3
classDef prominent fill:#c8a96e,stroke:#c8a96e,color:#1a1508,font-weight:600
classDef highlight fill:#2d2417,stroke:#c8a96e,stroke-width:1px,color:#e8d5a8

    subgraph InfraCore ["Infrastructure Engineering"]
        NLB["Network Load Balancer (Layer 4)"]:::prominent
        
        subgraph ComputeGrid ["Compute Pool: 12x VMs (40 OCPU, 128GB RAM, 53% Base Util)"]
            direction LR
            subgraph FD1 ["Fault Domain 1"]
                VM1["VM 1<br>40 OCPU<br>128GB RAM<br>50GB Boot Volume"]:::highlight
                VM2["VM 2"]:::steel
                VM3["VM 3"]:::steel
                VM4["VM 4"]:::steel
            end
            subgraph FD2 ["Fault Domain 2"]
                VM5["VM 5"]:::steel
                VM6["VM 6"]:::steel
                VM7["VM 7"]:::steel
                VM8["VM 8"]:::steel
            end
            subgraph FD3 ["Fault Domain 3"]
                VM9["VM 9"]:::steel
                VM10["VM 10"]:::steel
                VM11["VM 11"]:::steel
                VM12["VM 12"]:::steel
            end
        end
        
        subgraph DiskGrid ["VM 1 Network Disks: 8x Block Volumes (1525GB, 120 VPU)"]
            direction LR
            D1[("D1 Active<br>1525GB<br>120 VPU<br>2.68GB/s")]:::gold
            D2[("D2<br>Next")]:::steel
            D3[("D3<br>Next")]:::steel
            D4[("D4<br>Backup")]:::steel
            D5[("D5<br>Clean")]:::steel
            D6[("D6<br>Clean")]:::steel
            D7[("D7<br>Clean")]:::steel
            D8[("D8<br>Clean")]:::steel
        end
        
        DR["DR Rationale: 1 FD loss leaves 8 VMs at ~80% util.<br>9 VMs would spike to ~107% util (Cascade Crash)."]:::sage
    end

    NLB --> ComputeGrid
    VM1 -. "Attached Volumes" .-> DiskGrid
    ComputeGrid -. "Capacity Rules" .-> DR
    
    class InfraCore,ComputeGrid,FD1,FD2,FD3,DiskGrid ghost

Storage Disks:

Each VM is attached to 8x Elastic Network Block Volumes via iSCSI (multipathing enabled).

  • Disk Size: 1525 GB (1.5 TB) per disk.
  • Performance Tier: 120 VPUs per GB (Ultra High Performance).
  • Throughput Sweet Spot: OCI block volumes require a capacity of 1.5 TB to output their maximum performance limit of 350 MB/s of disk throughput.

4.3 Software Implementation Details

The Go telemetry application bypasses traditional file-writing functions and database protocols.

flowchart TB
    %% Compute Layer Golang Application Logic Diagram
    
    subgraph External_Ingress ["External Ingress"]
        NLB["OCI L4 NLB<br>(65 Gbps / 250k RPS)"]
    end

    subgraph Go_Node ["Golang Compute Node (Zero-Allocation Engine)"]
        
        subgraph Gnet_Ingestion ["gnet TCP Server"]
            MainReactor["gnet Main Reactor<br>(Accepts Connections)"]
            SubReactors["gnet Sub-Reactors<br>(1 per CPU core)"]
            Codec["Length-Prefix Codec<br>(4B Length + 32KB Payload)"]
            
            MainReactor -->|Distributes| SubReactors
            SubReactors -->|Stream| Codec
        end

        subgraph Memory_Management ["Memory Management"]
            SyncPool[("sync.Pool<br>(32KB Byte Arrays)")]
            Codec <-->|Borrow / Return| SyncPool
        end

        subgraph Write_Pipeline ["Write Engine (Zero-Backpressure)"]
            LockFreeAppend["Lock-Free Append<br>(atomic.AddInt64)"]
            Mmap["syscall.Mmap WAL<br>(100GB Chunks in RAM)"]
            
            Codec -->|Payload| LockFreeAppend
            LockFreeAppend -->|"copy()"| Mmap
        end

        subgraph Live_Read ["Live Tap (On-Demand)"]
            LiveSwitch{"isLiveClientActive?"}
            LiveChan["liveStreamChan<br>(Non-Blocking Broadcast)"]
            LiveClient["Live Client<br>(gRPC / WS)"]
            
            LockFreeAppend --> LiveSwitch
            LiveSwitch -->|True| LiveChan
            LiveChan -->|Zero I/O Cost| LiveClient
        end

        subgraph Archival_Offloader ["Background Offloader"]
            SealedChunk["Sealed 100GB Chunk"]
            UploadGoroutine["OCI Go SDK UploadManager"]
            S3[("OCI Object Storage<br>(Long-Term Archive)")]
            ResetChunk["madvise(MADV_DONTNEED)<br>Truncate & Reuse"]
            
            Mmap -->|Hits 100GB Boundary| SealedChunk
            SealedChunk --> UploadGoroutine
            UploadGoroutine -->|Stream| S3
            UploadGoroutine -->|200 OK| ResetChunk
            ResetChunk -.-> Mmap
        end

        subgraph Historical_Reads ["Historical Read Router"]
            Gateway["Query Aggregator / API Gateway"]
            TimeIndex["In-Memory TimeIndex<br>(B-Tree)"]
            
            Gateway -->|"Query (Timestamp)"| TimeIndex
            TimeIndex -->|Recent Data| Mmap
            TimeIndex -->|"Old Data (404)"| Gateway
            Gateway -.->|Redirect| Trino["OCI Big Data / Trino"]
            Trino -.->|Fetch| S3
        end

    end

    subgraph Hardware_Storage ["Hardware Layer"]
        OS_PageCache["Linux Kernel Page Cache<br>(msync / fdatasync)"]
        BlockVolumes[("4x OCI Block Volumes<br>(Ring Buffer Architecture)")]
        
        Mmap <-->|Asynchronous Flush| OS_PageCache
        OS_PageCache <--> BlockVolumes
    end

    %% Connections
    NLB -->|Raw TCP| MainReactor
    
    %% Styling
    classDef gold      fill:#1f1c15,stroke:#c8a96e,stroke-width:1.5px,color:#e8d5a8
    classDef sage      fill:#131a14,stroke:#7a9e7e,stroke-width:1px,color:#a8c8ac
    classDef steel     fill:#131620,stroke:#8896a8,stroke-width:1px,color:#b0bece
    classDef crimson   fill:#1a1212,stroke:#a87878,stroke-width:1px,color:#d4a0a0
    classDef ghost     fill:#111110,stroke:#2a2926,stroke-width:1px,color:#6a6860,stroke-dasharray:4 3
    classDef prominent fill:#c8a96e,stroke:#c8a96e,color:#1a1508,font-weight:600
    
    class NLB,LiveClient,S3,Gateway,Trino steel;
    class SyncPool,Mmap,TimeIndex sage;
    class BlockVolumes prominent;
    class MainReactor,SubReactors,Codec,LockFreeAppend,UploadGoroutine gold;
    class OS_PageCache ghost;

1. Ingestion Layer (gnet)

  • We use the gnet library (an event-loop networking framework) in raw TCP mode instead of HTTP.
  • Multi-Reactor Setup: The Main Reactor intercepts incoming connections and hands them off to Sub-Reactors. We pin one Sub-Reactor to each CPU core, preventing context switches.
  • Protocol Framing: We use a length-prefix format: [4-byte Length Integer] [32 KB Payload]. The framework buffers data until a complete 32,772-byte payload is assembled before triggering processing.

2. Zero-Allocation Memory Engine

  • sync.Pool: Slices are managed via sync.Pool to avoid memory allocation churn.
  • Memory-Mapped File (mmap): The Go app maps the active block volume directly into userspace memory via syscall.Mmap. Payloads are appended directly to this memory array.
  • Lock-Free pointer reservation: Multiple request threads write to the mapped memory concurrently without locking. When a payload arrives, the thread atomically reserves 32,780 bytes (32,768-byte payload + 12-byte WAL record header containing an 8-byte nanosecond timestamp and a 4-byte CRC32 checksum):
offset := atomic.AddInt64(&RingBuffer.Offset, 32780)

This reserves a slice in the mmap buffer. The thread copies the payload into its reserved space using Go's built-in copy() function.

  • Kernel Flushes: To prevent page cache saturation, the application periodically issues syscall.Msync / fdatasync calls, flushing data to disk in controlled, sequential blocks.

4.4 Ingestion & Storage Lifecycle

The memory-mapped Write-Ahead Log (WAL) operates as a state machine rotating across the attached network block volumes.

graph TD
%% ── Dark Elegant Theme ─────────────────────────────────
classDef gold      fill:#1f1c15,stroke:#c8a96e,stroke-width:1.5px,color:#e8d5a8
classDef sage      fill:#131a14,stroke:#7a9e7e,stroke-width:1px,color:#a8c8ac
classDef steel     fill:#131620,stroke:#8896a8,stroke-width:1px,color:#b0bece
classDef crimson   fill:#1a1212,stroke:#a87878,stroke-width:1px,color:#d4a0a0
classDef ghost     fill:#111110,stroke:#2a2926,stroke-width:1px,color:#6a6860,stroke-dasharray:4 3
classDef prominent fill:#c8a96e,stroke:#c8a96e,color:#1a1508,font-weight:600

    App["App Logic<br>(State Machine & Monitor)"]:::steel
    ObjStore[("OCI Object Storage")]:::sage

    subgraph Disks ["Elastic Network Disks (8x Total per Node: Active, Next, Backup)"]
        DiskA["Disk Set A<br>(Active Append)"]:::gold
        DiskB["Disk Set B<br>(Frozen / Next)"]:::steel
        DiskC["Disk Set C<br>(Backup Buffer)"]:::steel
        DiskE["Emergency Disks<br>(Provisioned On-Demand)"]:::crimson
    end

    %% Normal Lifecycle
    DiskA -- "1. Fills to 95% -> Pivot" --> DiskB
    DiskB -- "2a. Offload Success" --> ObjStore
    ObjStore -. "Clear Disks & Reuse" .-> DiskA

    %% Blocked Lifecycle
    DiskB -- "2b. Offload Delayed & Fills -> Pivot" --> DiskC
    DiskC -- "3. Active hits 60% & Offload Blocked -> Pivot" --> DiskE
    
    %% Application Intervention
    App -. "Provisions & Mounts" .-> DiskE
    App -. "Terminates Post-Recovery" .-> DiskE

1. Ingestion:

Telemetry data is appended to the active memory-mapped block volume (Disk A).

2. The Live Read Bypass:

When a live dashboard client connects, the Go app activates a boolean flag and opens a non-blocking Go channel. Ingestion threads write copy references directly to this channel, streaming data to live clients from RAM with zero disk read I/O.

3. Rotation State:

When Disk A reaches 95% capacity:

  • The Go app updates the atomic pointer, routing new write operations to Disk B.
  • Disk A is frozen. A background worker uploads the frozen 100 GB segment to OCI Object Storage using the OCI SDK UploadManager.
  • Upon upload completion, the application runs madvise(MADV_DONTNEED) to clear the RAM pages. The file is truncated, clearing the disk for reuse.
  • Disk C acts as a backup buffer if Object Storage upload speeds lag.

4. Dynamic Storage Provisioning:

If Object Storage uploads are delayed and active disk utilization reaches 60% with all buffer disks filled, the Go application calls the OCI API to provision and mount an emergency block volume. This prevents system crashes during object store slowdowns. Once network conditions normalize and the backlog is cleared, the emergency volumes are detached and terminated.

4.5 Historical Query Routing

To query historical data, we separate recent and cold data paths:

  1. Gateway Broadcast: Query requests (specifying a timestamp range) hit a Gateway Aggregator, which broadcasts the query to the nodes in the target cell.
  2. In-Memory B-Tree: Each Go node maintains a lightweight, in-memory B-Tree index mapping timestamps to byte offsets for the active volumes. It also tracks the OldestLocalTimestamp currently stored on its block volumes.
  3. Local Read Path: If the queried range is newer than OldestLocalTimestamp, the node locates the offsets in its local B-Tree, reads the exact byte slice from the block volume (+1 MB read), and streams it back to the Gateway.
  4. Cold Read Path: If the range is older than OldestLocalTimestamp, the node returns a 404 Not Local response. The Gateway then routes the query to OCI Big Data Service (Trino), which queries the data archived in OCI Object Storage.

4.6 Single Node Load Verification

To verify performance, we analyzed VM VNIC bandwidth limits against the maximum workload:

graph TD
%% ── Dark Elegant Theme ─────────────────────────────────
classDef gold      fill:#1f1c15,stroke:#c8a96e,stroke-width:1.5px,color:#e8d5a8
classDef sage      fill:#131a14,stroke:#7a9e7e,stroke-width:1px,color:#a8c8ac
classDef steel     fill:#131620,stroke:#8896a8,stroke-width:1px,color:#b0bece
classDef crimson   fill:#1a1212,stroke:#a87878,stroke-width:1px,color:#d4a0a0
classDef ghost     fill:#111110,stroke:#2a2926,stroke-width:1px,color:#6a6860,stroke-dasharray:4 3
classDef prominent fill:#c8a96e,stroke:#c8a96e,color:#1a1508,font-weight:600
classDef highlight fill:#2d2417,stroke:#c8a96e,stroke-width:1px,color:#e8d5a8

    subgraph Analysis ["Peak Network Load per VM (Single Node Analysis)"]
        NLB["Internet / NLB Ingress"]:::prominent
        Consumer["Consumer (Live Read)"]:::sage
        
        subgraph VM_Node ["Single VM (40 OCPU / 128GB RAM)"]
            direction TB
            NIC["Virtual NIC<br>(Full Duplex: 40Gbps Rx / 40Gbps Tx)"]:::highlight
            Processor["Go App (Data Processor & RAM Buffer)"]:::steel
            NIC <--> Processor
        end
        
        subgraph Disks ["8x Elastic Network Disks (Block Volumes)"]
            direction LR
            Active[("Active Disk<br>(Write Target)")]:::gold
            Draining[("Draining Disk<br>(Read Source)")]:::sage
            Idle[("6x Idle / Clean<br>Disks")]:::steel
        end
        
        ObjStore[("OCI Object Storage<br>(Archive)")]:::sage
        
        %% Traffic Flows (Concurrent)
        NLB -- "Inbound (Rx): Live Ingest<br>680 MB/s" --> NIC
        NIC -- "Outbound (Tx): Disk Write<br>680 MB/s" --> Active
        NIC -- "Outbound (Tx): Live Read (from RAM)<br>680 MB/s" --> Consumer
        
        Draining -- "Inbound (Rx): Disk Read (Drain)<br>2,000 MB/s" --> NIC
        NIC -- "Outbound (Tx): Object Storage Upload<br>2,000 MB/s" --> ObjStore
    end
    
    subgraph Justification ["Infrastructure Engineering Justification"]
        direction TB
        CalcRx["Receive (Rx) - Data entering the VM:<br>+ 680 MB/s: Live payload from LB<br>+ 2,000 MB/s: Read frozen disk via iSCSI<br>Total Rx: 2,680 MB/s<br>VM Limit: 5,000 MB/s (40 Gbps)<br>Headroom: 2,320 MB/s (Extremely safe margin)"]:::ghost

        CalcTx["Transmit (Tx) - Data leaving the VM:<br>+ 680 MB/s: Write payload to active disk via iSCSI<br>+ 680 MB/s: Serve Live Read from RAM to consumer<br>+ 2,000 MB/s: Upload frozen data to OCI Object Storage<br>Total Tx: 3,360 MB/s<br>VM Limit: 5,000 MB/s (40 Gbps)<br>Headroom: 1,640 MB/s (Comfortable safety margin)"]:::ghost
        
        Infra["Network Scale Justification:<br>Advertised 40Gbps VM network is full duplex (40Gbps Rx / 40Gbps Tx), giving 80Gbps total capacity.<br>This shape is mandatory to handle the concurrent 3,360 MB/s (Tx) peak load without throttling.<br>Global Scale: 4 Cells × 12 VMs = 48 VMs.<br>Storage Scale: 48 VMs × 8 Disks = 384 Network Disks."]:::gold
    end
    
    Analysis -.-> Justification
    
    class Analysis,VM_Node,Disks,Justification ghost

  • Ingress (Rx) Load:

    • Ingress Telemetry (from NLB): 680 MB/s680\text{ MB/s} (5.44 Gbps5.44\text{ Gbps}).
    • iSCSI Disk Read (for S3 Offload): 2,000 MB/s2,000\text{ MB/s} (16.0 Gbps16.0\text{ Gbps}).
    • Total Peak Rx: 2,680 MB/s2,680\text{ MB/s} (21.44 Gbps21.44\text{ Gbps}).
    • VM Rx Limit: 5,000 MB/s5,000\text{ MB/s} (40 Gbps40\text{ Gbps}).
    • Available Headroom: 2,320 MB/s2,320\text{ MB/s} (18.56 Gbps18.56\text{ Gbps}).

  • Egress (Tx) Load:

    • iSCSI Disk Write (Ingestion): 680 MB/s680\text{ MB/s} (5.44 Gbps5.44\text{ Gbps}).
    • Live Reader Stream (from RAM): 680 MB/s680\text{ MB/s} (5.44 Gbps5.44\text{ Gbps}).
    • Object Storage Upload: 2,000 MB/s2,000\text{ MB/s} (16.0 Gbps16.0\text{ Gbps}).
    • Total Peak Tx: 3,360 MB/s3,360\text{ MB/s} (26.88 Gbps26.88\text{ Gbps}).
    • VM Tx Limit: 5,000 MB/s5,000\text{ MB/s} (40 Gbps40\text{ Gbps}).
    • Available Headroom: 1,640 MB/s1,640\text{ MB/s} (13.12 Gbps13.12\text{ Gbps}).

This verification proves that a VM with a 40 Gbps NIC can handle peak read and write loads concurrently.

4.7 Host OS & Kernel Tuning Script

To maintain the required 250,000 RPS per cell, the OS must be tuned via standard configuration scripts during provisioning.

#!/usr/bin/env bash
# High-Performance OS Tuning Script for Telemetry Ingestion Node

# 1. CPU & Interrupt Tuning
# Set performance CPU scaling governor
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Disable irqbalance to prevent cache-miss latency spikes
systemctl stop irqbalance
systemctl disable irqbalance

# Manually pin interrupts: Cores 0-7 for VNIC/iSCSI queues, Cores 8-39 for Go App threads
# (Targeted core mapping applied to local interface queues)

# 2. File Descriptors & Process Limits
cat <<EOF >> /etc/security/limits.conf
* soft nofile 1048576
* hard nofile 1048576
root soft nofile 1048576
root hard nofile 1048576
EOF

# 3. TCP & Network Stack Configuration
cat <<EOF > /etc/sysctl.d/99-telemetry-engine.conf
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 300000
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.ipv4.tcp_mtu_probing = 1
net.ipv4.ip_local_port_range = 1024 65535
EOF
sysctl -p /etc/sysctl.d/99-telemetry-engine.conf

# 4. Memory & Block Device Configuration
# Disable Transparent Huge Pages (THP) to avoid memory latency overhead
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

# Tune attached iSCSI volumes (sdX)
for dev in /sys/block/sd*/queue/scheduler; do
    echo none > "$dev" 2>/dev/null || echo noop > "$dev" 2>/dev/null
done

for dev in /dev/sd*; do
    # Increase queue requests capacity
    echo 1024 > /sys/block/$(basename "$dev")/queue/nr_requests 2>/dev/null
    # Disable read-ahead
    blockdev --setra 0 "$dev" 2>/dev/null
done

5. Financial Ledger & ROI Verification

Below is the monthly cost breakdown for the state-machine ring buffer architecture on OCI compared against equivalent deployments on AWS.

5.1 OCI Monthly Infrastructure Cost

  • Compute Instances: 48x VM.Standard.A1.Flex (40 OCPU, 128 GB RAM):

48×$383.45=$18,405.6048 \times \$383.45 = \$18,405.60

  • Boot Volumes: 48x 50 GB Boot Volumes:

48×$2.13=$102.2448 \times \$2.13 = \$102.24

  • Elastic Block Volumes: 384x 1525 GB Disks (120 VPU @ $350 each):

384×$350.00=$134,400.00384 \times \$350.00 = \$134,400.00

  • OCI Base Infrastructure Total: $152,907.84
  • Flat-Rate Network Egress: OCI FastConnect unmetered 400 Gbps port: $20,700.00
  • Total Monthly Operational Expenditure: $173,607.84

5.2 Comparative Cost Analysis (OCI vs. AWS)

Expense Item AWS Managed Architecture State-Machine Ring Buffer (OCI) Monthly Savings
Compute & Cache $990,000 $18,405 $971,595
Ingestion Buffers (SSD/RAM) $1,900,000 (Scylla Ring) $134,400 (Elastic Volumes) $1,765,600
Data Egress (Internet) $4,200,000 (Metered Egress) $20,700 (FastConnect Port) $4,179,300
Inter-AZ Transfer $1,120,000 (Cross-AZ Replication) $0 (Unmetered VCN) $1,120,000
Object Storage (M1 Archive) $1,764,000 (Standard S3) $836,000 (OCI Storage) $928,000
Total Monthly Baseline $9,974,000 $1,009,505 $8,964,495

(Note: Storing 84 PB per month is the primary cost driver across both options. Mitigating this long-term requires volume discounts or data compression strategies, which are outlined in the future roadmap).

graph TD
    %% ── Classes ─────────────────────────────────────────────
    classDef gold      fill:#1f1c15,stroke:#c8a96e,stroke-width:1.5px,color:#e8d5a8
    classDef sage      fill:#131a14,stroke:#7a9e7e,stroke-width:1px,color:#a8c8ac
    classDef steel     fill:#131620,stroke:#8896a8,stroke-width:1px,color:#b0bece
    classDef crimson   fill:#1a1212,stroke:#a87878,stroke-width:1px,color:#d4a0a0
    classDef ghost     fill:#111110,stroke:#2a2926,stroke-width:1px,color:#6a6860,stroke-dasharray:4 3
    classDef prominent fill:#c8a96e,stroke:#c8a96e,color:#1a1508,font-weight:600

    %% ── Main Header ─────────────────────────────────────────
    TotalCost(["Total Monthly Infrastructure Cost: ~$152,907.84<br/>With Month 1 OCI Object Storage: ~$988,907.84"]):::prominent

    %% ── Cost Breakdown ──────────────────────────────────────

    subgraph PerNode ["1. Per Node Cost ($3,185.58)"]
        direction TB
        NodeVM["VM (40 OCPU / 128GB RAM)<br/>$383.45"]:::steel
        BootVol["Boot Volume<br/>$2.13"]:::sage
        NetDisks["8x Network Disks (1525GB, 120 VPU @ $350 each)<br/>$2,800.00"]:::gold
        
        NodeVM --- BootVol --- NetDisks
    end

    subgraph PerCell ["2. Per Cell Cost ($38,226.96)"]
        direction TB
        NodeCount["12x Nodes (VM + 8 Disks + Boot Vol)<br/>$38,226.96"]:::crimson
        CellDesc["Supports 250k RPS<br/>Distributed across 3 Fault Domains"]:::ghost
        
        NodeCount --- CellDesc
    end

    subgraph GlobalCost ["3. Global 4-Cell Cost ($152,907.84)"]
        direction TB
        CellTotal["4x Cells (48 VMs, 384 Disks)<br/>$152,907.84"]:::crimson
        GlobalDesc["Supports 1 Million RPS Global Traffic"]:::ghost
        
        CellTotal --- GlobalDesc
    end

    subgraph NetworkCost ["4. Network Egress"]
        direction TB
        Egress["84 PB Monthly Egress<br/>via FastConnect Partner<br/>~$20,000 - $30,000<br/>(Not included in total)"]:::sage
    end

    subgraph StorageCost ["5. Object Storage (Cumulative)"]
        direction TB
        StorageM1["Month 1 Capacity (at $0.01/GB)<br/>$836,000"]:::crimson
        StorageM23["Month 2: ~$1.6M<br/>Month 3: ~$2.5M"]:::crimson
        Discount["Requires Deep Enterprise Discount<br/>(Expected 30% - 50% reduction)"]:::ghost
        
        StorageM1 --- StorageM23 --- Discount
    end

    %% Connections from Total
    TotalCost --> PerNode
    TotalCost --> PerCell
    TotalCost --> GlobalCost
    TotalCost --> NetworkCost
    TotalCost --> StorageCost

6. Engineering Trade-offs & Challenges

By shifting complexity from off-the-shelf databases to a custom engine, we accepted specific architectural trade-offs:

graph TD
%% ── Dark Elegant Theme ─────────────────────────────────
classDef gold      fill:#1f1c15,stroke:#c8a96e,stroke-width:1.5px,color:#e8d5a8
classDef sage      fill:#131a14,stroke:#7a9e7e,stroke-width:1px,color:#a8c8ac
classDef steel     fill:#131620,stroke:#8896a8,stroke-width:1px,color:#b0bece
classDef crimson   fill:#1a1212,stroke:#a87878,stroke-width:1.5px,color:#d4a0a0
classDef ghost     fill:#111110,stroke:#2a2926,stroke-width:1px,color:#6a6860,stroke-dasharray:4 3
classDef prominent fill:#c8a96e,stroke:#c8a96e,color:#1a1508,font-weight:600

    Title["Architecture 2 Hidden Trade-offs<br>(Trading Database CapEx for Elite Engineering OpEx)"]:::prominent

    subgraph Challenges ["The 5 Critical Flaws of a Custom 'Do-It-Yourself' Database"]
        direction TB
        
        C1["1. Engineering & Maintenance Trap<br>OS/Kernel friction with rapid disk mounts.<br>Bus Factor of 1-2 (Requires elite low-level systems engineers)."]:::crimson
        
        C2["2. The 'Failsafe' Flaw (Cloud APIs)<br>OCI APIs take 30-90s to mount emergency disks.<br>At 680 MB/s, this forces a ~40GB+ RAM buffer.<br>High risk of Out-of-Memory (OOM) application crashes."]:::crimson
        
        C3["3. Node Death & Data Recovery<br>No native replication or clustering protocol.<br>Node crashes leave 'dirty' orphaned disks behind.<br>Requires manual intervention or fragile custom recovery scripts."]:::crimson
        
        C4["4. Split-Brain Read Nightmare<br>Data is fractured across 4 states at any given moment:<br>Live RAM → 48 Local Disks → In-Flight → Object Storage.<br>Scatter-gather API queries will cause unpredictable read latency."]:::crimson
        
        C5["5. Hidden Financial Tax (Object Storage PUT APIs)<br>1M RPS = 2.59 Trillion requests/month.<br>Unbatched 32KB PUTs to OCI = Millions of $$$ in API fees.<br>Forced large batching (64-128MB) increases data loss risk on crash."]:::crimson
    end

    Title --> C1
    Title --> C2
    Title --> C3
    Title --> C4
    Title --> C5
    
    class Challenges ghost

6.1 The Engineering & Maintenance Trap

Developing a proprietary storage engine increases technical debt. The system requires low-level engineering expertise in Linux kernel internals, virtual memory layout, and system calls (mmap, madvise, msync). This narrow skill requirement increases recruitment costs and limits team scale.

6.2 Cloud API Provisioning Latencies

Using the OCI API to attach emergency block volumes takes 30 to 90 seconds. At a write speed of 680 MB/s680\text{ MB/s}, a delay of 90 seconds requires the Go application to buffer up to 61.2 GB61.2\text{ GB} of data in RAM (680 MB/s×90 s=61.2 GB680\text{ MB/s} \times 90\text{ s} = 61.2\text{ GB}). If the VM’s memory footprint is mismanaged, this buffer can trigger Out-of-Memory (OOM) kernel panics.

6.3 Recovery after Node Failure

Because the custom engine does not replicate data at the block layer, a node crash leaves its block volumes orphaned in an un-flushed, dirty state.

  • Recovery Flow: A recovery script detects the node failure, detaches the orphaned block volumes, and attaches them to a new VM.
  • Index Rebuild: The new node reads the raw headers from the disk to reconstruct the B-Tree index, offloads the data to OCI Object Storage, and is then terminated. This restores the system state without data loss, but requires maintaining custom orchestration scripts.

6.4 Split-Brain Read Routing

Because telemetry data is transiting across multiple states (Live RAM, local block volumes, in-flight uploads, and OCI Object Storage), query aggregation requires a routing mapping. The Query Gateway must resolve where target data segments are located, which can cause latency spikes for queries crossing storage tier boundaries.

6.5 API Call Cost Management

Uploading unbatched 32 KB payloads to OCI Object Storage would generate high API fees. To manage costs, the Go application buffers data into 100 GB segments before issuing PUT requests. This reduces transaction fees but increases the amount of un-flushed data buffered in memory.

7. The Future Horizon: C++ Seastar, DPDK, SPDK, and DAOS

As our telemetry network scales from 1,000,000 RPS to 10,000,000 RPS (representing a write load of 320 GB/s320\text{ GB/s}), the physical limits of OCI network interfaces will become a bottleneck. We have designed a future architectural roadmap to scale beyond these limits.

flowchart LR
%% ── Dark Elegant Theme ─────────────────────────────────
classDef gold      fill:#1f1c15,stroke:#c8a96e,stroke-width:1.5px,color:#e8d5a8
classDef sage      fill:#131a14,stroke:#7a9e7e,stroke-width:1px,color:#a8c8ac
classDef steel     fill:#131620,stroke:#8896a8,stroke-width:1px,color:#b0bece
classDef crimson   fill:#1a1212,stroke:#a87878,stroke-width:1px,color:#d4a0a0
classDef ghost     fill:#111110,stroke:#2a2926,stroke-width:1px,color:#6a6860,stroke-dasharray:4 3
classDef prominent fill:#c8a96e,stroke:#c8a96e,color:#1a1508,font-weight:600

    subgraph External ["External Ingress (1M RPS)"]
        direction TB
        GlobalRouter{"DNS / Global LB"}:::gold
        NLB["OCI Network Load Balancer<br>(L4 / Direct Routing)"]:::prominent
        GlobalRouter --> NLB
    end

    subgraph ComputeNode ["C++ Compute Tier (DPDK/Seastar Shared-Nothing)"]
        direction TB
        NIC["OCI 100GbE NIC Hardware<br>(SR-IOV / VFIO-PCI)"]:::steel
        Mbuf["Hugepages Memory<br>(rte_mempool / mbuf)"]:::sage
        
        subgraph Cores ["Thread-Per-Core (Pinned CPUs)"]
            direction TB
            Core1["Core 1: DPDK Rx Ring -> Userspace TCP -> App Logic"]:::gold
            Core2["Core 2: DPDK Rx Ring -> Userspace TCP -> App Logic"]:::gold
        end
        
        SPDK["SPDK Appender<br>(Direct Local NVMe)"]:::steel
        
        NIC -- "DMA Write" --> Mbuf
        Mbuf -. "Pointer Ref" .-> Cores
        Cores -- "Pass mbuf (Zero Context Switch)" --> SPDK
    end

    subgraph Storage ["Storage Fabric (RDMA / RoCEv2)"]
        direction TB
        DAOS_API["DAOS API (libdaos)<br>Async RDMA Push"]:::prominent
        DAOS_Pools[("DAOS Distributed Container<br>(Key-Array 32KB Objects)")]:::gold
        S3[("OCI Object Storage<br>(Native Tiering / Archive)")]:::sage
        
        DAOS_API --> DAOS_Pools
        DAOS_Pools -- "Zero-Copy Async Flush" --> S3
    end

    %% Main Data Flow (Left to Right)
    NLB -- "Raw Frames<br>(RSS Hashing by IP/Port)" --> NIC
    SPDK -- "spdk_thread<br>(Bypass Kernel)" --> DAOS_API
    
    %% Optional layout adjustments to "make it go around" / visually balanced
    S3 -. "Metadata / Tiering Status" .-> GlobalRouter

    class External,ComputeNode,Cores,Storage ghost

7.1 DPDK-Driven Userspace Networking

  • To bypass the Linux kernel TCP/IP stack, we will transition the engine to C++ using the Data Plane Development Kit (DPDK).
  • By binding the VNICs to DPDK userspace drivers (vfio-pci), the application gains direct access to network card ring buffers via Direct Memory Access (DMA). This avoids kernel context switching, page faults, and system call overhead.
  • We will run a userspace TCP stack pinned to individual CPU cores, using Receive Side Scaling (RSS) to distribute packets without CPU core synchronization.

7.2 SPDK-Driven Direct Block Storage

  • To bypass the kernel filesystem layer and SCSI drivers, we will use the Storage Performance Development Kit (SPDK).
  • SPDK uses userspace drivers to communicate with NVMe block volumes via PCIe.
  • This allows the application to write data blocks directly to raw storage sectors without kernel block device scheduling, reducing write latencies to single-digit microseconds.

7.3 DAOS-Based Storage Fabric

  • For persistent storage scaling, we will implement Distributed Asynchronous Object Storage (DAOS).
  • DAOS uses Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCEv2) networks. This allows nodes to read and write to shared storage pools without CPU intervention, enabling scalable data replication and tiering.

8. Lessons Learned & Conclusion

  1. Understand Platform Economics: High-throughput system architecture must be designed alongside cloud billing structures. Selecting a cloud provider based on network and storage pricing (like OCI's egress structure) can be as critical as code efficiency.
  2. Respect Mechanical Sympathy: At scale, general-purpose software frameworks can introduce significant overhead. Aligning application design with hardware characteristics—such as matching OCI's OCPU count to VNIC bandwidth limits and using mmap for disk writes—is essential to maximize performance.
  3. Optimize System Interfaces: Managing large data ingestion requires tuning system-level parameters. Implementing Jumbo Frames (MTU 9000), using Event-Loop reactors, and disabling kernel interrupts are necessary steps to prevent operating system bottlenecks.

True systems architecture requires designing within the bounds of both software physics and cloud financial constraints. By replacing generic database layers with a custom storage engine, we met our performance goals while keeping operational costs within target budgets.

The Impact

We transitioned the infrastructure to Oracle Cloud (OCI) and architected a lock-free, zero-allocation ring buffer. We eliminated the database bottleneck via direct memory-mapped network disks, achieving deterministic scaling while slashing operational expenditures to under $175,000 monthly.

Run this measurement against your own system.

Deployment Audit: £500, fixed scope. Credited in full against the next stage.

What this costs
Back to Case Studies