Dynatrace is set to change the data management underpinnings of its cloud-based AIOps tool, starting with new support for log analytics that could set the stage for new competition with Splunk, Elastic, Sumo Logic and others.
The new architecture, called Grail, culminates more than three years of engineering work behind the scenes, Dynatrace officials said. Grail includes a massively parallel processing engine, a new query language and a data lakehouse, so named because it combines elements of both a data lake for fast ingesting unstructured data and a data warehouse for fast processing of structured data. This type of hybrid architecture is also marketed for general purpose data analytics used by vendors such as Snowflake and Databricks.
In Dynatrace’s case, Grail was specifically built to handle the data that feeds the company’s AIOps products for observation, meaning log analytics data that can now be stored directly alongside events and traces information that Dynatrace already manages based on its application performance management (APM) roots.
“Grail is a massively parallel processing engine for data management,” said Rick McConnell, CEO at Dynatrace. “The first use of this, we will apply to log management and analytics, because our view is that the log market with Splunk, Elastic and others has so far remained siled and very isolated.”
Dynatrace log analytics: Third time’s the charm?
This isn’t Dynatrace’s first attempt at integrating log analytics into its AIOps tools, said Gregg Siegfried, an analyst at Gartner. In February 2021, it added log data support to its Software Intelligence Platform, and in September of that year, SpectX was acquired to add additional log analytics features. But by February 2022, some users expressed frustration after their first attempts to link Dynatrace’s log monitoring tools to back-end BI systems.
“It’s going to be bigger than just the logs, but that’s kind of the space for them,” Siegfried said.
BT Digital, a multi-national telecom division based in London, signed on to replace dozens of separate IT monitoring tools and ticketing systems with a combination of ServiceNow and Dynatrace products in June. The company is about 25% of the way through this conversion project, covering nearly 80,000 virtual hosts in all. BT has yet to test Grail — but one of the telecom’s IT leads hopes Grail will further reduce the number of separate IT management tools his company uses.
“Assuming this proves itself in a proof of concept, what this means for us is we have a single place to go for log analytics and root cause analysis,” said Alex Bell, ServiceNow delivery and engineering lead at BT. “We think there are three or four more [log management and analytics tools] which we can only now bring under a single ability.”
Overall, BT has an ambitious set of goals for its AIOps at Dynatrace, up to and including what it calls ZeroOps, also known in the industry as NoOps, where systems become automatically self-healing without intervention of man.
At the moment, BT is starting AIOps automation for relatively simple cases, such as server restarts, but the ability to store more data for Dynatrace analysis could help the effectiveness of the automated root cause analysis and event correlation under AIOps in the future, Bell said.
“The more confident you are in that root cause, the faster you can get to the root cause and the more automated you can get there, the more effective the ZeroOps vision will be for us,” he said.
More data, more competition
Dynatrace will introduce a new proprietary query language, Dynatrace Query Language (DQL), when Grail becomes available to SaaS customers next month, which vendor executives claim will improve SaaS performance , economics and integrations with third-party data analytics systems, including third-party business analytics and BI tools.
That query language is based on a graph API similar to the open source GraphQL API that has begun to gain momentum in advanced data management circles. VMware is taking its own approach to tools that includes a knowledge graph in its Aria portfolio launched in late August; a cloud-native SBOM tool in the works from CNCF will include a graph database.
Knowledge graphs and graph APIs are gaining popularity in IT automation and management tools due to the increasing complexity of relationships between microservice apps, where knowledge graphs can make inferences about faster than relational and NoSQL databases queried via the REST API.
Dynatrace already has a knowledge graph feature it calls Smartscape, but Grail and DQL will allow for more custom, high-performance queries of multiple types of data within the same storage repository, rather than need to bridge between separate data pools as Smartscape currently does , said Steve Tack, senior vice president of product management at Dynatrace.
“[Customers] can bring their own business logic, their own queries, in a more independent way [for] use cases [such as] fraud detection that we don’t provide out of the box with Dynatrace,” said Tack. “But now we give them access to data from logs, traces, user experience sessions and more, and they’ll be able to create their own use cases.”
Pricing has yet to be set for Grail-based services, but the application of Grail’s massively parallel data processing engine will also mean it can handle data ingestion for one-tenth the cost of competitors , McConnell said. Dynatrace also plans to price Grail based on the number of queries, rather than the amount of data stored or users with access, he said.
While Dynatrace is clearly focused on competitors such as Splunk, Elastic and Sumo Logic, it will need to demonstrate Grail’s effectiveness before it can unseat established log analytics competitors, Siegfried said.
“Splunk Enterprise effectively created the log analytics market as we know it today, and they have the next generation [observability] product at SignalFx, but they haven’t proven that they know how to sell application performance monitoring,” he said. “At the same time, application logs are a part of what Splunk deals with — they also deal with logs for security and IT operations that aren’t necessarily connected to an application — and it’s unclear whether the Grail aspect of Dynatrace will be positioned as that kind of general-purpose log analysis tool.”
Both Splunk and Dynatrace have focused on large top-down enterprise sales for observability tools, while rival Datadog has also begun to encroach on their turf with a bottom-up, developer-focused approach, which is landing its own big deals with the likes of Mendix, and showing strong growth in recent earnings reports.
“Datadog is firing on all cylinders, both on the product development side and on the sales and marketing execution side,” Siegfried said. “Dynatrace tends to focus on bigger companies and bigger deals and Datadog usually does the land-and-expand thing on a team basis, but they’re starting to get noticed at the executive level as well.”
Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. He can be reached at [email protected] or on Twitter @PariseauTT.