DataFrame Explorer
Last year Chartio came up with the idea of Visual SQL Editor which is great. It is great for people who are creating the charts. When I stared on the journey of SQL Frames last year, while I liked what Chartio has done, I felt something was missing. Especially if you look at the licensing of most BI products, which say 5 creators and unlimited viewers or something along those lines, what's missing becomes a bit more obvious. It is that the Visual SQL Editor only caters to those 5 creators (and may be a few more) but not the plenty of end users consuming those charts and dashboards.
So, what can be done to all those users? This has been a quest for SQL Frames and I think the answer may be what I call as DataFrame Explorer™.
Dev Productivity
Over the last couple of decades, the people working on digital transformation have created tools and techniques not only for end users but for themselves to be better at their jobs.
Declarative
Specifiying logic declaratively is one example of dev productivity. Rather than repeatedly writing code telling how to do the computation, it is expressed in a domain specific language (DSL), and a special program converts (or interprets) the logic expressed with the DSL to something suitable for execution.
For data, SQL is an example of one such DSL. In spite people tried to replace SQL with various alternatives it stood the test of time and that is because it is declarative. If there is something that is wrong about SQL, it is perhaps the syntax and once someone masters it or there are plenty of tools to help with simple things like auto-completition, then SQL is infact quite productive compared to someone doing the same things procedurally.
Stack Frames
Ask any developer, how valuable a debugger is. One thing great about debuggers is they allow you to setup a breakpoint and then navigate back into the execution stack to examine the various stack frames to understand what went wrong where. Debuggers are so helpful that as people moved from desktop applications to web applications, the corresponding application code became very complex that browsers started offering debuggers for JavaScript.
Time travel debugging
In recent years, another concept that became popular with single page application frameworks such as React is the time travel debugging. That is, an application is composed of several components and each component has its own state. This state is constantly changing and if something goes wrong, people want to know how they ended up in that state. The time travel debugging allows examing the history of the state changes to isolate the problem.
User Productivity
The above are some of the examples of developer productivity enhancing tools. Great, but end users are far more than developers. What about productivity enhancing tools for them? Yes, there are many such tools as well. Any well designed UX is a great example.
When it comes to data, contextual information, relevant information, timely information and consice information are all means of enhancing productivity. Many SAAS applications are already heading towards that direction, if not already. However, much of that effort these days is focused towards collaboration and sharing. Sure you can send things via Slack and keep all your team in loop with a single click. But how many of them are able to undestand what you sent and are able to explore further themselves for better insights?
User Enablement
Productivity tools help automate repetitive and mundane tasks. This helps users of these tools to spend more time on more advanced tasks which can be creative and/or complex. However, this requires a different set of tools to enable users to perform such tasks. In fact, the productivity tools of developers described above, repurposed for end user tasks are exactly the type of tools that can enable these users to perform complex tasks.
SQL Frames is meant to offer such tools, to enable end users to not just share their report but also explore the report and all related data from source to the final table or chart following all the data transformations along the way. Just like a developer walks through the runtime stack to isolate issues or a react developer uses the time travel debugging, a regular end user provided with any key metric can explore all data associated with that metric following each and every step of the transformation to understand better and make the right decisions. Introducing, DataFrame Explorer to achieve this.
Conclusion
The pandemic has seen unprecedented levels of embracing remote work whether the management believed in it or not. This resulted in many companies providing collaboration tools. But I think there is an opportunity to provide user enablement tools so these remote workers don't feel lost (their work-from-office colleagues can just give a shout or walk down the aisle to their colleagues for help).
SQL Frames DataFrame Explorer is one such tools for user enablment for anyone working with data in your organization. And these days, that should be almost everyone.
What Chartio started off with Visual SQL Editor is a productivity tool for data developers (data scientists, data engineers, data analysts collectively referred as data developers). SQL Frames is bringing that ambition to its natural conclusion by providing a digital user enablement tool for anyone dealing with data with DataFrame Explorer™.