Databricks More Complex Up Front but Significantly Cheaper than Snowflake

ETR Insights presents an interview with the Senior VP of Global Infrastructure for a large financial services organization. This executive highlights a shift from the early post-COVID IT spending surge to a more moderated expenditure strategy, emphasizing workforce attrition, automation, and bulk software purchasing. The organization is intensively investing in AI and machine learning to enhance decision-making and operational efficiency; they use Microsoft Purview to analyze employee interactions, maintain a diverse database environment, and prefer Databricks over Snowflake for data warehousing. Read on to learn why this company favors simple tools like Excel and ClickUp for project management, and why they feel observability tools are not always worth the money.

Vendors Mentioned: Amazon / Atlassian (Confluence) / ClickUp / Databricks / ExtraHop / Gigamon / KnowBe4 / MariaDB / Microsoft (Azure Purview, Copilot, Excel, SharePoint, SQL Server) / MySQL / Palo Alto Networks (Cortex XDR) / Snowflake

Vendor-Specific Commentary

Databricks > Snowflake. This organization leans heavily on Databricks – in both Amazon and Azure – as well as Azure Synapse. They are moving business units away from Snowflake towards Databricks. “Databricks is much more complex up front; we have to have a more skilled data engineer managing it versus a data analyst, but it’s much more powerful, at least in our case. And then once we start scaling, the Databricks price was significantly cheaper than Snowflake.” The company prefers a strategy of investing in a few skilled individuals, seeking depth in talent over numbers for their Databricks operations.

Microsoft – Proceeding with Caution. While the company is reviewing Copilot, their current focus is on Purview, in particular ways to analyze employee interactions for potential harassment. Our guest notes Microsoft’s broader strategy of refining offerings based on market response, as opposed to pioneering innovation. They are wary of committing to Copilot, favoring solutions that seamlessly integrate across diverse ecosystems over more insular offerings from Microsoft. “We’ve already invested in other platforms outside the Microsoft ecosystem, and a lot of times the Microsoft solutions only work with Microsoft products, or they’re cumbersome to integrate.”

Databases. The company uses a wide variety of databases, from Microsoft SQL Server to MariaDB and MySQL, depending on product needs. This diverse database lineup is here to stay, driven by vendor specifications when purchasing software, cost disparities among database solutions, and the financial and labor challenges of rearchitecting databases.

Project Management. Our guest has a lean approach to project management. They primarily utilize Excel alongside ClickUp, a tool whose extensive feature set they barely tap into but appreciate for its notification and task-assignment capabilities. Atlassian’s footprint is reduced, in part due to the company’s transition from on-premises solutions to cloud-only offerings coupled with a more aggressive pricing model. Microsoft’s alternatives present a more cost- effective solution that meets most of their needs. “Confluence from Atlassian is a far superior product, hands down, but I can do probably 75% of that in SharePoint for the cost of the license I’m already paying for.” Our guest theorizes that Microsoft’s promotion of the .NET stack is part of its appeal to large enterprises, while Atlassian seems more tailored for startups and smaller entities.

Information Security. Our guest favors a simplified security vendor landscape, minimizing integration challenges that could arise with a mishmash of best-of-breed offerings. They are heavy users of Palo Alto Networks. “I think they continue to improve. When you acquire company after company after company to grow – and they’ve been great purchases – but sometimes they don’t integrate them as well as it could be. [But] they’ve fixed some of that.” Our guest warns against an outdated “castle moat” approach, which emphasizes external threats while often overlooking vulnerabilities within the organization. They highlight the importance of user education and tools like KnowBe4. “Without that, we can just keep spending money on tools and playing whack-a-mole.” The emphasis should be on making training more engaging, favoring brief, interactive weekly sessions over lengthy videos and quizzes.

Data Analytics. While they are happy that their relational databases have achieved stability and don’t demand considerable attention, our guest looks to enhance their analytics capabilities. “In our world, we feel like the better we understand the data, the more informed business decisions the leaders can make. That’s really where IT can help move the needle for the business.”

Observability. The aim is for developers to possess a robust understanding of company applications; our guest questions the cost-value proposition of expensive monitoring tools. “I think these tools popped into the market and a bunch of companies jumped on and said, ‘I can solve a problem we’re having.’ But then over time they’re looking at how expensive these tools are and questioning why am I paying a developer this amount of money when they don’t understand the product that they wrote? I think there’s a value to some of the tools, but there’s a big cost to them.”

IT Spending Trends – Looking to ML/AI to Increase Efficiencies and ROI

Our guest, reflecting on post-COVID IT spending, suggests that while there was an initial surge in expenditure to quickly transform businesses during the pandemic, there’s now a trend to taper that spending. They emphasize a combination of strategies: workforce attrition, automating employee-intensive processes, and leveraging bundled software purchasing deals to achieve bulk discounts. “We are working with partners to bundle and get group purchase options on software. Instead of getting the best-of-breed of every single product, maybe bundling with a single software vendor and getting a bulk discount on that.”

This executive’s technology priorities are centered around expanding their existing AI and machine learning capacities. “We’re invested in both of those, to make business decisions and help inform the business leaders on areas they could focus on.” New capital is being allocated to drive these technologies, aiming to streamline operations and increase efficiency. The ultimate goal is to balance the upfront investment with long-term returns, both in operational savings and improved financial products that increase revenue.

Request to Join

The ETR Community is an exclusive group of IT decision makers. If you have purview over the tech spend in your organization, please fill out the form below to see if you qualify to join the ETR Community. Ready to jump in quicker? Applications with LinkedIn links included are able to be reviewed more quickly. Thank you!