CMU Database Group - SQL or Death? Seminar Series (2025) Speaker: James Cowling (https://www.linkedin.com/in/jcowling) February 10, 2025 https://db.cs.cmu.edu/seminar2025/#db1 Sponsors: Google DAPA Team (https://google.com)

James is joined by Andy Pavlo, Associate Professor of Computer Science at Carnegie Mellon University, about the dramatic shifts reshaping the database industry in 2024. The conversation explores how Redis's controversial licensing changes sparked unprecedented community backlash and the fierce battle between Snowflake and Databricks that culminated in a $2 billion acquisition. Andy also goes into modern testing approaches, particularly deterministic simulation, and why database wars now extend beyond performance metrics. The conversation also touches on the evolution of database education, the rising dominance of Postgres, and how modern testing approaches are reshaping reliability expectations in database development and deployment. Check out Andy's 2024 retrospective on the world of databases at https://www.cs.cmu.edu/~pavlo/blog/2025/01/2024-databases-retrospective.html. The CMU Database Group has a ton of *excellent* educational content available online if you want to learn more about databases and how they work. One good starting point is the Intro to Database Systems lecture series https://youtu.be/otE2WvX3XdQ?si=jp_CFc_EGNqdellr.

A good set of values can be critically important in establishing a team that stands for something. This post outlines why company values are important...

In this episode of Databased, Tom Redman speaks with Jamie Turner and James Cowling about the complexities of database isolation levels and their implications for application development. They discuss the differences between isolation levels such as read uncommitted, read committed, repeatable read, and serializable, emphasizing how these choices can impact data integrity and performance. The conversation also highlights common anomalies that can arise from improper isolation level selection and the importance of understanding these pitfalls. Additionally, James and Jamie advocate for starting with strong guarantees to simplify development and ensure scalability as applications grow. Key Topics Discussed: - The significance of database isolation levels and how they affect data integrity and application performance in concurrent environments. - Common anomalies that can occur when using different isolation levels, including lost updates and dirty reads, and their impact on applications. - The importance of understanding the trade-offs between higher isolation levels and performance, especially in high-concurrency scenarios. - Best practices for application developers to start with strong isolation guarantees to simplify development and ensure data consistency. - The role of optimistic concurrency control in managing conflicts during transactions and its implications for user experience. - Strategies for architecting systems to minimize contention and optimize performance while maintaining data integrity. - The historical context of databases and distributed systems merging, leading to confusion in terminology and definitions. - Real-world examples of how isolation levels can lead to unexpected results in applications, emphasizing the need for careful design. Key Takeaways: - Understand the different database isolation levels — read uncommitted, read committed, repeatable read, and serializable — to make informed decisions for your application’s data integrity. - Identify potential anomalies that can arise from using various isolation levels, such as lost updates and dirty reads, to prevent unexpected behaviors in your application. - Start with strong isolation guarantees in your database configuration to simplify development and ensure robust data consistency as your application scales. - Implement optimistic concurrency control techniques to manage transaction conflicts effectively and enhance user experience during concurrent operations. - Design your database schema to minimize contention by ensuring that independent transactions do not interfere with each other, improving overall performance. - Evaluate the performance trade-offs of higher isolation levels against your application’s needs to determine the most suitable approach for your use case. - Utilize select for update statements in transactions to lock rows and prevent concurrent modifications that could lead to data inconsistencies. - Monitor your application’s performance and data integrity regularly to identify and address any issues related to isolation levels and transaction behavior. - Educate your development team on the nuances of database isolation levels and their implications to foster better decision-making in application design. - Reassess your architecture periodically as your application grows, adapting your isolation level and concurrency strategies to meet evolving performance requirements.

In this episode of Databased, Jamie Turner and James Cowling explore critical principles of systems design, emphasizing the importance of preparing for worst-case scenarios rather than best-case outcomes. They dive into the concept of congestion collapse, illustrating how systems can fail under pressure and the need for robust designs that maintain performance during peak loads. Additionally, they discuss the significance of achieving zero errors in data systems, highlighting strategies for implementing verification processes to ensure data integrity. Tune in to gain valuable insights on building resilient systems that can withstand challenges and support long-term growth! Key Topics Discussed: - How designing systems for worst-case scenarios prevents congestion collapse and ensures reliability under peak load conditions. - Why congestion collapse occurs when systems fail to handle increased requests, leading to widespread service outages. - The importance of implementing back-off strategies for retries helps mitigate the impact of excessive load on system performance. - How understanding the state of your system is crucial for maintaining operational efficiency and preventing unexpected failures. - How achieving zero errors in data systems enhances team velocity and reduces the need for troubleshooting and maintenance. - Why verification jobs are essential for continuously checking data integrity and ensuring consistent system performance over time. - How prioritizing strong guarantees in system design simplifies development and allows for easier optimization as needs evolve. Key Takeaways: - Design for worst-case scenarios to ensure your system remains reliable and resilient under peak load conditions. - Implement back-off strategies for retries to prevent congestion collapse and reduce unnecessary load on your system. - Monitor system state continuously to identify potential issues before they escalate into significant failures or outages. - Establish verification jobs that regularly check data integrity to maintain zero errors in your data systems. - Document and communicate your system's guarantees and invariants to ensure all team members understand the expected behavior. - Prioritize simplicity in design by focusing on clear state definitions to make troubleshooting and optimization easier. - Conduct regular load testing to understand how your system behaves under stress and identify potential bottlenecks. - Encourage a culture of ownership among team members to proactively address inconsistencies and maintain data quality. - Utilize monitoring tools to track performance metrics and alert you to any deviations from expected system behavior. - Review and refine your retry logic to ensure it aligns with your system's capacity and prevents cascading failures.

The high level is that we’re pretty happy with the Convex API right now so the current swing within the company is to focus more on scale and polish for a while, with the caveat that we have some big features already in the works that are coming out soon.

In this episode of Databased, Tom Redman leads a discussion joined by Sujay Jayaker and James Cowling, two scientists behind Dropbox's synchronization protocol. They cover database synchronization in general, what it means alongside the local-first movement, and what went into designing sync at Convex. Talking points: - What is the local-first movement actually? - What are the primary use-cases for offline sync/local-first? - In a local-first world, how do you decide which version of state is correct? - What are different strategies for reconciling state? Is there a right one? - Is the local-first offline dream actually possible while sharing state externally? - Where does opt-in local-first excel? - What are the product and UX implications of inefficient sync protocols? - What role do transactions play in all of this? - Why clients directly talking to the database is a bad idea - Finally, what is sync?

In this episode of Databased, Tom Redman engages Jamie Turner and James Cowling in exploring the transformative impact of AI on software development, particularly through tools like Cursor AI. They discuss the balance between efficiency and the essential understanding of coding principles, emphasizing the risks junior developers face when relying too heavily on AI. Their conversation also dives into the distinction between prototype and production-ready applications, highlighting the importance of human creativity and problem-solving in engineering. They also discuss how AI can enhance productivity while ensuring that foundational skills remain vital in the ever-evolving tech landscape. Key Topics Discussed: - The role of AI tools like Cursor in enhancing developer productivity and efficiency during the software development process. - Concerns about junior developers over-relying on AI, potentially hindering their foundational learning and problem-solving skills. - The critical distinction between prototype applications and production-ready software, emphasizing the importance of thorough testing and scalability. - Human creativity remains essential in software engineering, ensuring that technology complements rather than replaces human expertise and insight. - The need for a strategic approach to using AI in development, incorporating layers of understanding and oversight. - The impact of AI on the future job landscape for software engineers, particularly regarding junior and senior roles. - Ongoing discussions about how AI can assist in coding while still requiring deep technical knowledge and conceptual understanding.

In this episode of Databased, Jamie Turner and James Cowling grapple with some bad news… they think each other’s product sucks…? Jamie and James candidly discuss the challenges and misconceptions surrounding Convex. For example, they explore the learning curve for new users, emphasizing the balance between conceptual understanding and practical application. The conversation also touches on the potential for developer lock-in and the importance of building trust with users. Jamie and James also explore the intricacies of software development and user experience, including the challenges of transparent pricing models, emphasizing the need for relatable cost estimations based on user requirements. Additionally, they highlight strategies for enhancing customer support and education, ensuring users can fully leverage the platform's capabilities. By acknowledging their product's shortcomings, they aim to foster a more transparent dialogue within the software development community. Key Topics Discussed: - The initial learning experience with Convex, such as the time it takes for developers to grasp new concepts and methods. - The difference between conceptual understanding and practical application, emphasizing the importance of both in mastering Convex. - Concerns about developer lock-in, and how specialization in Convex might affect career mobility and skill transferability. - The significance of opinionated frameworks that guide users, helping them make informed decisions while enhancing productivity. - The balance between providing flexible tools and maintaining structured guidelines to support users as their projects evolve. - The necessity of clear pricing structures, allowing users to estimate costs based on their specific application needs and user growth. - The importance of robust customer support, ensuring users can navigate the platform effectively and maximize its benefits. - The critical role of indexing in database management, emphasizing the need for efficient queries to optimize performance. - The value of case studies to illustrate real-world applications and challenges, making technical concepts more relatable to users. - The importance of educating users on writing efficient SQL queries, helping them avoid performance pitfalls in their applications. - Implementing feedback forms to gather user insights and critiques, fostering continuous improvement of the platform based on user experiences. - Incorporating positive affirmations and recognition within the development process to motivate teams and enhance productivity. Key Takeaways: - Despite the initial learning curve, developers can become productive with Convex within a few days of use. - The importance of viewing oneself as a software engineer rather than just an expert in a specific tool, even Convex. - The significance of establishing trust with users, especially when introducing a new product in a competitive market. - The value of feedback from the developer community, which helps them identify areas for improvement in Convex. - Releasing Convex may inadvertently seed competition, highlighting the dynamic nature of the software development landscape. - Implementing opinionated frameworks in software design to guide users toward making informed decisions and enhancing overall productivity. - Achieving balance flexibility and structure in tools to accommodate evolving project needs while providing clear guidelines for users. - Developing transparent pricing models that allow users to easily estimate costs based on their specific application requirements and anticipated growth. - Enhancing customer support by creating comprehensive resources and training materials that empower users to navigate the platform effectively. - Optimizing query performance by focusing on indexing strategies that improve the efficiency of SQL queries and overall database management. - Creating a pricing simulator to help users visualize potential costs based on different scenarios, aiding in budget planning and decision-making. - Utilizing case studies to demonstrate real-world applications platform, making technical concepts more relatable and understandable for users. - Educating users on SQL query writing to help them avoid common performance pitfalls and improve their data analysis capabilities.

Convex is a BaaS (backend-as-a-service) platform, built to abstract away many of the backend tasks that developers have to deal with when starting a new project. It provides standard BaaS elements including a database, vector search and compute, but differentiates itself by replacing SQL queries with standalone function-calling - which enables developers to run code directly in their database. The startup’s mission is to help developers move fast and differentiate themselves where it matters - in their application itself, and not the backend.

Traditional wisdom is to never bet against SQL but we’re here to convince you otherwise. The app development ecosystem accretes complexity year over year as a result of decades of bad design patterns and attempts to solve problems on the wrong layers of the stack. Consistent caching, dynamic subscriptions, and end-to-end type safety can all be provided automatically when databases are built with the right abstractions: queries that are real imperative programs. We’ll give an overview of the problems facing modern application developers and how these can be solved by a better database. We’ll also dive into some of the details and challenges when implementing this stack from the ground up at Convex. https://x.com/jamesacowling/ https://www.linkedin.com/in/sujayakar/ Talk from Systems Distributed '24: https://systemsdistributed.com Join the chat at https://slack.tigerbeetle.com/invite

In this episode of Databased, Tom Redman speaks with Jamie Turner and James Cowling about the historical context that necessitated the invention of SQL 1.0 and how these databases have scaled over time. They touch on the challenges of maintaining transactionality and consistency, the rise and limitations of NoSQL databases, and optimizing for developer velocity without compromising on foundational database principles. This episode is a must-listen for anyone interested in database technology and its legacy as it is carried into modern software development. Key Topics Discussed: - Evolution of SQL databases from early implementations to modern-day MySQL and Postgres. - Importance of transactionality and consistency in relational database management systems. - Challenges of database scaling and performance optimization for high concurrency and large datasets. - Comparison between SQL and NoSQL databases, highlighting strengths and limitations. - The role of SQL as the lingua franca for querying databases. - Strategies for balancing developer velocity with robust database management. - Future trends and innovations in database technology, focusing on maintaining core principles while adapting to new demands. Key Takeaways: - The historical context and evolution of SQL databases to appreciate their foundational role in data management. - The transactionality and consistency in database design to ensure data integrity and reliability. - The strengths and limitations of SQL versus NoSQL databases when choosing the right database for your application. - Query performance by using indexing and proper query planning to handle high concurrency and large datasets. - Balancing developer velocity with robust database practices to maintain sustainable growth and development efficiency. - SQL as a powerful querying language that simplifies complex data retrieval tasks. - The benefits to adopting modern database solutions that incorporate the core principles of relational models while addressing new technological demands. - Current database practices to identify areas where transactionality and consistency might be compromised. - Strategies to scale databases effectively, keeping in mind performance and reliability. Quotes: “I don't think SQL was the key innovation in the history of databases. The relational database management system, the RDBMS, that was the innovation. And SQL was a declarative language that one would use to query an RDBMS. So at the time, now, this is 50 years ago, this is before even the concept of abstraction in programming languages was invented.” “SQL is just one manner of expressing a desire to fetch certain data. You could also just write that code by hand if you wanted. And so, yes, relational database systems are a very fundamental, big idea. And SQL in some ways is it's a very popular means of giving a command to a relational database management system.” And I think what's interesting about that is most things are, you know, actually not technical. At least most things succeed that to any degree in the market. They're usually socioeconomic and things like that. And the thing that started changing, I think that was correlated with the rise of the software developer as the most important labor type, the most important worker type in the technology industry, right?” “I mean, startups and software teams starting a new project very justifiably want to optimize for velocity, and that is generally the right choice. But I think you have to ask, when you're optimizing for velocity, are you doing so in a sustainable way?”

In the second episode of Databased, Tom Redman speaks with Jamie Turner and James Cowling about the evolution of single-page applications (SPAs) and their roller-coaster-like relevance in the industry. They weigh in on the debate between SPAs and multi-page applications (MPAs), and how server-side rendering (SSR) impacts SEO. The trio also shares practical insights on optimizing web performance, emphasizing the balance between innovation and real-world application. Whether you're a developer or tech enthusiast, this episode offers valuable perspectives on the future of web development. Key Topics Discussed: -The evolution and current relevance of single-page applications in the web development landscape. -The debate between single-page applications and multi-page applications for modern web projects. -The impact of server-side rendering on search engine optimization and user experience. -Practical insights on optimizing web performance and balancing innovation with real-world application. -Historical perspective on early web development techniques, including CGI bin scripts and static websites. -The significance of reducing page load times and its effect on user engagement and interaction. -The role of server components in simplifying web development and improving application performance. -Real-world examples and case studies illustrating the benefits and challenges of different web architectures. -The importance of maintaining modular and ergonomic code for better development practices. -Predictions on the future trends in web development and the potential impact of new technologies. Key Takeaways: -Evaluate the pros and cons of single-page applications versus multi-page applications for your specific project needs. -Implement server-side rendering to improve search engine optimization and enhance user experience. -Incorporate modular and ergonomic code practices to improve maintainability and scalability of your web applications. -Utilize server components to simplify development and enhance application performance. -Measure the impact of reducing page load times on user behavior and conversion rates. -Adopt best practices for balancing innovation with real-world application to achieve practical and effective web solutions. -Explore the benefits and challenges of different web architectures through real-world examples and case studies. Quotes: “jQuery is very inspiring. It was like, well, you maybe could actually turn this into something, but I don't think it went far enough on having, you know, opinionated patterns that really simplified your world.” “So the optimizations that may, that were taking on the complexity at Meta scale, it almost definitely is not worth taking on the complexity at your scale. See, those optimizations do not matter to you yet.” “Far more simply, you shouldn't use a distributed database if you don't need a distributed database, right? And so just because Google has all these amazing infrastructure engineers working there does not mean that's a good role model for how to design systems.” “Also, something about the way people are starting to talk about this too, just reminds me of like the, like a refinement is less sexy than a revolution, and it doesn't retweet as well. So probably what we need is a refinement, not really a revolution. And if it was a true revolution, it probably wouldn't just look like a repackaging of old ideas we already left behind.”

Even though Convex supports the creation of cron jobs out of the box, until now, we've only supported static jobs. With user space crons, we now suppo...

In this inaugural episode of Databased, Tom Redman speaks with Jamie Turner and James Cowling, diving deep into the world of edge computing and its impact on modern web development. Key Topics Discussed: -What is "the edge" in computing, and why has it become a buzzword? -The evolution of edge computing from useful technology to overhyped solution -Physical limitations of distributed systems, including the speed of light -Challenges of maintaining data consistency across distributed locations -Use cases where edge computing makes sense vs. where it adds unnecessary complexity -The importance of co-locating compute with data for most applications -How platforms can leverage edge technology without burdening developers Key Takeaways: -Edge computing isn't inherently bad, but it’s often not the right solution for most applications -The laws of physics (speed of light) create fundamental limitations for distributed systems -Co-locating compute with data often provides better performance than edge computing for data-intensive applications -Developers should focus on building features quickly rather than prematurely optimizing with complex distributed systems -Platforms should abstract away the complexities of edge computing, allowing developers to focus on building their applications Quotes: #1.) "Edge computing was touted as the ultimate solution, but we've hit a reality check—it's not the cloud, and it's not a miracle cure." – Tom Redman #2.) "The idea of having a server closer to you than your mailbox sounded great, but the reality is most of our data is still sitting in Virginia." – Jamie Turner #3.) "When hype runs up against the laws of physics, you get growing pains. Edge computing is facing that exact challenge." – James Cowling #4.) "Ultimately, the source of truth for data has to live in a single physical location. Distributed systems are complex and often not worth the trade-off." – James Cowling

In this episode, James Quick and Amy Dutton chat with James Cowling, co-founder of Convex, about designing infrastructure for product engineers. James explains the innovative features of Convex, including its JavaScript-based queries and real-time data subscriptions, and compares it to Firebase. They also discuss the challenges of edge computing, the importance of user state, and the role of AI in modern development.

Convex is a serverless backend platform to simplify fullstack application development. Its underlying database is written in Rust, and it uses TypeScript to integrate with reactive UI frameworks. The platform is growing, which has presented new reasons to make the code open source, and Convex recently released the source code for a self-managed version of their platform.

Our CTO James gave a talk at Developer Week 2024 about how backends have been letting down product developers by not providing clean abstractions that empower development without introducing unnecessary complexity. James outlines four major failures of the status quo: * Bad ideas in query models * Bad ideas in type systems * Bad ideas in caching and performance * Bad ideas in data sync He explains why these exist, how they are holding back developers, and discusses recent innovations at Convex that address these four areas. 0:20 - Intro and academic context 1:27 - Lessons to learn from Barbara Liskov 2:12 - The productivity crisis of programming 4:25 - The platform crisis of programming 5:53 - Bad ideas in querying databases 8:56 - Good ideas in querying databases 9:55 - Bad ideas in type systems 11:46 - Good ideas in type systems 13:06 - Premature formality 14:53 - Bad ideas in caching and performance 16:23 - Good ideas in caching and performance 17:32 - Bad ideas in data syncing 18:37 - Good ideas in data syncing 21:12 - The solution to the platform crisis

Learn how to automatically retry actions in Convex while also learning a little about scheduling, system tables, and function references.

In this episode, Nathan sits down with three members of the a16z x Convex AI Town project: Yoko Li (Partner, a16z), Martin Casado (GP, a16z), and James Cowling (CTO, Convex). AI Town is a virtual town where AI agents live, interact, and socialize. They discuss how AI Town originated from Yoko’s companion app project, unpredictability as a feature in LLMs and interacting with models like they are lifeforms, and why they chose Javascript and Convex to build AI Town. If you're looking for an ERP platform, check out our sponsor, NetSuite: http://netsuite.com/cognitive 📣 CALL FOR FEEDBACK: To borrow from a meme… we're in the podcast arena trying stuff. Some will work. Some won't. But we're always learning. http://bit.ly/TCRFeedback Fill out the above form to let us know how we can continue delivering great content to you or sending the feedback on your mind to [email protected]. TIMESTAMPS: (00:00:00) - Episode Preview: Intro to AI Town and the idea of AI as companions (00:05:29) - Overview of AI Town and the simulation framework based on the Stanford Generative Agents paper (00:08:24) - Yoko explains how the idea for AI Town originated from the companion app project (00:10:41) - Yoko discusses how she built the initial AI Town prototype and wanted to make it multiplayer (00:12:31) - The simplicity and elegance of the AI Town codebase (00:13:52) - Interacting with LLMs is like interacting with lifeforms (00:15:47) - Sponsors: Netsuite | Omneky (00:18:25) - How Convex built a server-side game engine for AI Town (00:19:28) - How Convex makes building a game engine easy with transactions and database support (00:23:39 )- James emphasizes the power of functional programming paradigms like Convex for building AI apps (00:25:02) - Using simple JavaScript so anyone could understand and extend AI Town (00:28:39) - The group reflects on how JavaScript has become so powerful compared to languages like C++ (00:30:23) - How AI coding assistants were used in building AI Town (00:31:22) - No open source code for the Stanford paper when they started (00:33:25) - The interplay between programmer and AI model (00:38:01)- Martin draws a distinction between using formal languages vs. natural language (00:39:52) Unpredictability as a feature in LLMs (00:43:21) The balance between formal language and unpredictable behaviours in LLMs (00:43:59) AI Town’s future and the beauty of the community (00:48:27) Are we living in a simulation? (00:50:38) Advice for other developers in AI (00:54:29) AI Town is a community project to be extended on LINKS MENTIONED: - AI Town (try it out!): https://www.convex.dev/ai-town - AI Town Github: https://github.com/a16z-infra/ai-town - Convex: https://www.convex.dev/ - The Sea of Tranquility: https://www.goodreads.com/en/book/show/58446227 - Generative Agents, Interactive Simulacra of Human Behaviour: https://arxiv.org/abs/2304.03442 X/TWITTER: @stuffyokodraws (Yoko) @martin_casado (Martin) @jamesacowling (James) @realaitown (AI Town) @labenz (Nathan) @eriktorenberg @CogRev_Podcast SPONSORS: NetSuite | Omneky NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. Music Credit: GoogleLM Music license: 00UR6KUNCXJCJWMU

👍🏼 Give your thumbs-up and subscribe to the channel! 🎙️More Software Engineering Daily podcasts: softwareengineeringdaily.com ⌚Timestamps:

We host James Cowling, co-founder & CTO at Convex, and former Senior Principal Engineer at Dropbox. At Convex, James is building a backend platform that reinvents how apps are developed by putting together functions, a database, file storage, scheduling, search, and realtime updates. Previously, he led design, implementation and launch of Dropbox's multi-exabyte geo-distributed storage system, along with the migration of Dropbox data from Amazon S3 to in-house infrastructure. Here is what we talked about 👇 00:01:02 Planning software and uncertainty 00:04:45 Knowns and unknowns 00:10:10 Stepping stones 00:17:41 Management & estimates 00:24:50 Project variables 00:31:26 Psychology for engineers 00:35:48 Optimize your career for having fun 00:41:35 From engineer to manager 00:48:42 Keeping tech + business aligned in hyper-growth

Today, a new wave of innovation is making databases approachable for developers without requiring a specialized skillset, opening the door to a whole new category of ambitious web experiences. In this panel, you’ll hear from the people pushing the boundaries of database technology and learn what’s possible — and what’s coming. 00:00 Introduction 01:22 SQL or no SQL? 09:38 Explaining the concept of managed databases 11:23 The greatest pains for database developers 21:41 How to choose the right database for your project 27:42 What would you like to see in the future of databases?

Convex 0.5.0 introduces built-in support for actions — arbitrary lambda functions that run in a Node.js environment on the Convex cloud. We use the te...

The serverless movement has eliminated so many problems but it has left application developers with the hardest one of all: managing distributed state...

The future of databases is Convex. https://convex.dev This video originally aired at Jamstack 2022, highlighting the next evolution of modern web development architecture. Source code for the demo is available at https://github.com/JamesCowling/swaghaus

No engineering leader is capable of sitting down and charting a straight-line path to success on a complex project. Real engineering is an iterative process of delivering incremental value while illuminating unknown unknowns. The solution to executing on complex projects without over-engineering, over-resourcing, over-designing and overly risk-taking is usually re-scoping the project into a carefully formulated set of stepping stones. Try Sentry: https://sentry.io Docs: https://docs.sentry.io Follow: https://twitter.com/getsentry

James Cowling is the co-founder of Convex, a state management platform for web developers. We discuss the state of web development in the industry today, and the various different approaches to make it easier.

Convex makes storing your application’s data as easy as using React state management. If you can use React hooks, you can also manage your backend data using Convex. James Cowling is a former Dropbox infrastructure engineer turned startup founder. James joins the show to discuss how Convex offers a simpler full-stack developer experience than the

Dynamic application developers need global state management, not a database. James (@jamesacowling) presents his talk from the HYTRADBOI 2022 database conference on how Convex is rethinking how app developers manage global state. See the full HYTRADBOI schedule at hytradboi.com. Join the Convex beta at https://convex.dev.

Often the biggest impacts a Staff Engineer can make in their organization are not technical but rather people-related. When teams are value-aligned due to good leadership, they go on to make larger impacts than they would otherwise have. As Senior...

CMU Database Group - Vaccination Database Tech Talks - Second Dose (2021) Speakers: James Cowling (Convex) November 22, 2021 https://db.cs.cmu.edu/seminar2021-dose2#db12 Sponsors: OtterTune (https://ottertune.com) Steven Moy Foundation for Keeping it Real (https://stevenmoyfoundation.org)

When is it a good time to start using Rust? The founding team at Convex has had the privilege of leading development on some of the most heavily used Rust-based systems in the world: * Magic Pocket [https://dropbox.tech/infrastructure/extending-magic-pocket-innovation-with-the-first-petabyte-scale-smr-drive-deployment] , Dropbox's geo-distributed data storage system. This system has run

From Zig SHOWTIME #28 https://zig.show Original papers: http://pmg.csail.mit.edu/papers/vr.pdf http://pmg.csail.mit.edu/papers/vr-revisited.pdf Find James on Twitter: https://twitter.com/jamesacowling 0:00 Intro 2:25 Brian Oki 26:11 James Cowling

Dropbox was the first major tech company to deploy SMR (Shingled Magnetic Recording) technology at scale, migrating a significant fraction of it's multi-exabyte storage system to an append-only design targeted to SMR drives. Dropbox was able to leverage SMR to save over 20% in cost compared to its previous generation of storage technology, with no performance impact for end users. Dropbox engineer and prior Storage Team lead James Cowling will talk about the journey Dropbox took to deploy SMR,the architectural changes that were required to switch to an append-only storage model, and future avenues for optimization within the storage stack.

Structure projects around delivering incremental value and eliminating unknown unknowns.

Avoid engineering bias by orienting a team around a mission not a system.

Allow a team to be more productive by making mistakes.

Being a strong tech lead is very different from being a strong engineer. Let’s start by dispelling a common leadership myth: that you should “lead by example.”

How to estimate the durability guarantees of a storage system. There will be math!

Chapter 17. Engineering for Data Durability James Cowling, Dropbox, Inc. SREs live and breathe reliability, but to many engineers the word reliability is synonymous with availability: “How do we keep the site up?” Reliability is a multifaceted concern, however, and an extremely important part of this is durability: “How do we avoid losing or corrupting our data?” Engineering for durability is of paramount importance for any company that stores user data. Most companies can survive a period of downtime, but few can survive losing a significant fraction of user data. Building expertise in durable systems is particularly challenging, however; most companies improve availability over time as they grow and as their systems mature, but a single durability mistake can be a company-ending event. It’s therefore important to invest effort ahead of time to understand real-world durability threats and how to engineer against them.

James Cowling, Principal Engineer at Dropbox, reviews how the company's infrastructure provides a competitive advantage in the marketplace. This includes efficiencies in cost, performance, and overall agility. Recorded at Storage Field Day in Silicon Valley on March 7, 2018. For more information, visit http://TechFieldDay.com/event/sfd15/

James Cowling, Principal Engineer at Dropbox, provides background on the development and technical details on their Magic Pocket, their custom built, internally hosted, software defined, cloud storage infrastructure. Recorded at Storage Field Day in Silicon Valley on March 7, 2018. For more information, visit http://TechFieldDay.com/event/sfd15/

James Cowling of Dropbox (architect of their distributed storage system), speaks to Robert Blumen about their move from Amazon’s S3 to their own infrastructure; The show covers: the size, scope and scale of Dropbox’s data management; their experience on Amazon’s S3; why S3 over time did not meet their needs; how the decision was made to build their own; the architecture of Magic Pocket; building the skill sets they needed but did not have at the time; are layer violations ever a good thing?; his team’s “zero errors” policy; the important of invariants; how devops (sre) made the move easier; how they migrated; a brief excursion into the hardware side; what can engineers learn from this migration that will apply to other migrations (even smaller ones)?

Aaron and Brian talk with James Cowling (@jamesacowling; Storage Team Lead @Dropbox) about the Dropbox migration from AWS, project “Magic Pocket”, building distributed systems, lessons learned and how much better something must be before making a massive change.

There is nothing more important to Dropbox than the safety of our user data. When we set out to build Magic Pocket, our in-house multi-exabyte storage system, durability was the requirement that underscored all aspects of the design and implementation. In this post we’ll discuss the mechanisms we use to ensure that Magic Pocket constantly maintains its extremely high level of durability.

We started the day with James Cowling from Dropbox talking about how the company’s exabyte-scale system protects all of its documents. Instead of describing the system architecture, which he wrote about in a blog post, he talked about what it takes to preserve data in the face of all kinds of faults — including system ...

Dropbox has been storing files on Amazon Web Services for 8 years, and Dropbox’s core business is storing files. For the past three years, Dropbox has been working on a project to migrate its file storage from Amazon Web Services to its own custom-built infrastructure. Magic Pocket is the name of Dropbox’s new infrastructure layer,

We’ve received a lot of positive feedback since announcing Magic Pocket, our in-house multi-exabyte storage system. We’re going to follow that announcement with a series of technical blog posts that offer a look behind the scenes at interesting aspects of the system, including our protection mechanisms, operational tooling, and innovations on the boundary between hardware and software. But first, we’ll need some context: in this post, we’ll give a high level architectural overview of Magic Pocket and the criteria it was designed to meet.

How to engineer against data loss. Selection from Velocity Conference New York 2015

This paper presents an updated version of Viewstamped Replication, a replication technique that handles failures in which nodes crash. It describes how client requests are handled, how the group reorganizes when a replica fails, and how a failed replica is able to rejoin the group. The paper also describes a number of important optimizations and presents a protocol for handling reconfigurations that can change both the group membership and the number of failures the group is able to handle.

This thesis presents Granola, a transaction coordination infrastructure for building reliable distributed storage applications. Granola provides a strong consistency model, while significantly reducing transaction coordination overhead. Granola supports general atomic operations, enabling it to be used as a platform on which to build various storage systems, e.g., databases or object stores.

Despite the increasing importance of protecting confidential data, building secure software remains as challenging as ever. This paper describes Aeolus, a new platform for building secure distributed applications. Aeolus uses information flow control to provide confidentiality and data integrity. It differs from previous information flow control systems in a way that we believe makes it easier to understand and use. Aeolus uses a new, simpler security model, the first to combine a standard principal-based scheme for authority management with thread-granularity information flow tracking. The principal hierarchy matches the way developers already reason about authority and access control, and the coarse-grained information flow tracking eases the task of defining a program�s security restrictions. In addition, Aeolus provides a number of new mechanisms (authority closures, compound tags, boxes, and shared volatile state) that support common design patterns in secure application design.

This paper presents Granola, a transaction coordination infrastructure for building reliable distributed storage applications. Granola provides a strong consistency model, while significantly reducing transaction coordination overhead. We introduce specific support for a new type of independent distributed transaction, which we can serialize with no locking overhead and no aborts due to write conflicts. Granola uses a novel timestamp-based coordination mechanism to order distributed transactions, offering lower latency and higher throughput than previous systems that offer strong consistency.

We present Census, a platform for building large-scale distributed applications. Census provides a membership service and a multicast mechanism. The membership service provides every node with a consistent view of the system membership, which may be global or partitioned into location-based regions. Census distributes membership updates with low overhead, propagates changes promptly, and is resilient to both crashes and Byzantine failures. We believe that Census is the first system to provide a consistent membership abstraction at very large scale, greatly simplifying the design of applications built atop large deployments such as multi-site data centers.

Replicated state machines are an important and widely-studied methodology for tolerating a wide range of faults. Unfortunately, while replicas should be distributed geographically for maximum fault tolerance, current replicated state machine protocols tend to magnify the effects of high network latencies caused by geographic distribution. In this paper, we examine how to use speculative execution at the clients of a replicated service to reduce the impact of network and protocol latency. We first give design principles for using client speculation with replicated services, such as generating early replies and prioritizing throughput over latency. We then describe a mechanism that allows speculative clients to make new requests through replica-resolved speculation and predicated writes. We implement a detailed case study that applies this approach to a standard Byzantine fault tolerant protocol (PBFT) for replicated NFS and counter services. Client speculation trades in 18% maximum throughput to decrease the effective latency under light workloads, letting us speed up run time on single-client micro-benchmarks 1.08-19x when the client is co-located with the primary. On a macro-benchmark, reduced latency gives the client a speedup of up to 5x.

The current approach to developing rich, interactive web applications relies on asynchronous RPCs (Remote Procedure Calls) to fetch new data to be displayed by the client. We argue that for the majority of web applications, this RPC-based model is not the correct abstraction: it forces programmers to use an awkward continuation-passing style of programming and to expend too much effort manually transferring data. We propose a new programming model, MapJAX, to remedy these problems. MapJAX provides a shared memory abstraction between the browser and the server, based on the familiar primitives of objects, locks, and threads. MapJAX also provides additional features (parallel for loops and prefetching) that help developers minimize response times in their applications. MapJAX thus allows developers to focus on what they do best-writing compelling applications-rather than worrying about systems issues of data transfer and callback management.

There are currently two approaches to providing Byzantine-fault-tolerant state machine replication: a replica-based approach, e.g., BFT, that uses communication between replicas to agree on a proposed ordering of requests, and a quorum-based approach, such as Q/U, in which clients contact replicas directly to optimistically execute operations. Both approaches have shortcomings: the quadratic cost of inter-replica communication is unnecessary when there is no contention, and Q/U requires a large number of replicas and performs poorly under contention.

There are currently two approaches to providing Byzantine-fault-tolerant state machine replication: a replica-based approach, e.g., BFT, that uses communication between replicas to agree on a proposed ordering of requests, and a quorum-based approach, such as Q/U, in which clients contact replicas directly to optimistically execute operations. Both approaches have shortcomings: the quadratic cost of inter-replica communication is unnecessary when there is no contention, and Q/U requires a large number of replicas and performs poorly under contention.