We started Pangea in "interesting times" (October 2021), amidst COVID and all the anxiety around work from home, mental stress issues, social distancing, etc. Since then, there have been tremendous shifts in the technology industry. In just the last six months: tech has gone from a hyper-growth and crazy hiring sprees to massive layoffs. 2023 will be a year to watch as economic and global situations evolve. But often, it is these tough times that lead to the most interesting technological innovations.
We have embarked on a bold idea represented in the company name Pangea (a supercontinent that existed almost ~250 Million years ago) - one place for all your app’s security functions and use cases.
What is Pangea?
Today, there are thousands of security vendors. Most security products and services (offered as UI-first experiences) are there to plug the holes or weaknesses in your app, process, infrastructure, systems, etc. Pangea is a bottoms-up approach. We are building cost-effective, “pay-as-you-go” security microservices delivered as APIs to be embedded directly in your application, giving your apps a secure foundation on which to build. Pangea is on a mission to provide a wide variety of security services that are not usually the core expertise of development teams yet are a necessity. Pangea’s APIs completely remove the complexity of building, scaling, managing, and monitoring these security capabilities. We allow developers to focus on their applications’ core features, dramatically accelerating time-to-market while improving overall security.
We started off ideating, discussing, and listing many use cases and corresponding services that we could build. We engaged with our advisors and design partners, and quickly settled on a few valuable services that cover a range of complex capabilities. Doing so was a conscious decision that helped ensure that we built a scalable, flexible architecture that can handle the diverse requirements of future Pangea services.
Setting up for success
In terms of building the platform, we started writing production code in Dec 2021, with a few fundamental attributes that shall eventually define Pangea. Pangea has to be multi-cloud, multi-region, multi-tenant, secure, performant, and scalable.
Given that we are a startup with limited time and resources, we had to wisely architect and invest in a few key areas upfront to ease and accelerate building microservices. We quickly organized the team into a few functional groups:
Infrastructure: Responsible for orchestrating CI/CD pipelines, managing and monitoring the cloud resources, and enabling service authors to express resource requirements for use by services. Most importantly provide full visibility into the workings of the platform by implementing internal alerting and dashboarding for platform usage.
Platform: Responsible for providing the core framework for building microservices while being multi-CSP and multi-region. Platform also provides the abstraction layer, such that service authors can focus on business logic rather than the underlying infra layer. However, abstraction has to be cleverly managed via appropriate resource provisioning and configuration with no performance impact. Platform and Infra functions collectively provide common functionality around logging, tracing, metrics, etc. Platform is also responsible for core internal services like the API gateway, Metering, Billing, etc.
Microservices: Responsible for delivering many microservices and APIs that provide the security features to our customers, the app developers.
Console: Responsible for providing management and administrative access to the APIs, usage dashboards, service metrics, etc. Access to interactive and relevant documentation is another key function of this team.
Architect for the worst-case, implement for the best outcome
Once organized, we got right into key architectural decisions, prototyping, and production code. For the API Architecture, we very quickly agreed on JSON-RPC as it scales best to a variety of short-term and long-term use cases, variability in request parameters, and complex response payloads. The platform architecture must scale to known and future unknown use cases. Future-proofing is almost impossible, but to the best extent possible, the architecture needs to be flexible and accommodate future requirements that may arise. From a service resiliency perspective, we wanted to ensure that every API is serviced in the most efficient and performant way, with the worst-case outcome being performance degradation rather than outright failure. To fulfill this expectation, we architected a call routing approach:
with optimal performance for the majority of use cases
only sluggish in rare use cases
that practically never fails
Guarantees are almost impossible given that our assurances are at the mercy of cloud providers hosting our platform, but we have designed for the best user experience possible.
We leverage direct service-to-service communication for the best performance while using message queues as a fallback when services are scaling, updating, deploying, or unavailable for any reason. Our API gateway is key to scale and performance, for which we evaluated many offerings, but given the features and flexibility we needed, we decided to build it from scratch. The API gateway interfaces with many backend services - performing authentication and authorization (using Open Policy Agent - OPA, more on that in a future blog) and then efficiently routing incoming requests to the correct microservices to get responses as fast as possible. The API gateway and the inter-service communication architecture allows us to serve requests synchronously or asynchronously independent of the service REST handler implementation. Thus, the central theme of architectural choices and decisions has been to allow service authors to write services quickly, abstracting common challenges (call routing, metering, logging, etc.) away the same way that Pangea is endeavoring to abstract the complexity of security away from application developers.
Making the right choices
Building a Cloud Native API platform in 2022, when cloud providers have really matured, and many native services and technologies like K8s are stable mainstream technologies - is a sheer joy and a blessing. We had the luxury of choosing the most modern stack and tooling. However, this “cutting-edge” sword can swing either way, and we must continually make careful choices that set us up in the right direction and enable us to run fast. In the initial phases of a startup, when designing and building a product/platform, we have the flexibility to innovate, take risks, and bet on some of the most modern capabilities. However, we must learn from our failures, adjust our course, and pivot fast. Speed of execution, scale, performance, security, and usability are front and central to all our decisions. All the way from the structure of our REST endpoints (JSON-RPC) to automated documentation generation from OpenAPI specs, to automation in infra - every decision has been thoughtful, debated, and weighed carefully. And the fruits of our labor have started to pay off. The platform design has enabled us to roll out new production features in a matter of hours. I have seen larger companies take months just to get consensus and agreement; then implementation follows, and by the time they have something in front of their users, it's already too little too late.
Preparing for GA
As we have been getting ready for GA, we have been dealing with the neverending nervousness and dilemma of - “are we ready?” Is the product/platform complete enough? Does it provide good enough value to users? There is never a good answer but launching sooner is always better because there is no substitute to real user feedback. On some services, like the Secure Audit Log, we have gone quite deep with really strong tamperproofing features, leveraging blockchain technology for compliance (HIPAA, SOX, SOC, NERC-CIP etc.) needs and use cases. In other cases, the features have been less complex (like Redact service that removes PII data) while we wait for more user feedback before we cater to more advanced use cases like redact-on-read. We know and understand that maturity is a journey and not a fixed state.
GA at last!
With the best engineering team and thoughtful architecture, in about six months since we started writing production code, we launched Beta in July 2022. Since then, we have had 200+ users come and try our platform. And here we are in December 2022, ready with our GA release - complete on AWS, with multi-cluster, multi-region (US and EU), and multi-tenancy delivered as cloud-native platform and services. We have many microservices with more on the way, a great UI, beautiful dashboards, pricing calculator and pay-as-you-go billing. Not only that, but we enable developers to hit the ground running with SDKs in four languages, interactive documentation (that includes getting started guides, use cases, and sample code), and integrations with Firebase, Okta/Auth0. We top that off with a successfully completed SOC 2 Type 1 audit! All of this is delivered on a modern tech stack and infrastructure, and orchestrated via K8s and Terraform. Feel free to sign up here for a free account which comes with a $5 credit to try out the various APIs.
An exciting future awaits
We have learned a lot along the way, constantly incorporated the lessons, and strengthened our platform. We have only unveiled just five developer-facing services (Secure Audit Log, Redact, Embargo, File Intel, Domain Intel) and we are eager to unveil many more (AuthN, Vault, Secure Object Store, IP Intel, URL Intel, File Scan and AuthZ) that are in the last stages of testing and polish. Looking back in the rear view mirror, all that is no small feat. I could not be more proud of our awesome team who have accomplished this monumental task at such lightening speed!
This, and my upcoming blogs are intended to share what we have learned and built. It's kind of open-sourcing the architecture. I welcome and encourage you to share your similar experiences, as we look to improve, expand our catalog of microservices, and continue to build on the foundation of Pangea as the first ever SPaaS solution.
