The only thing faster than the speed of light is the expectation of a massive IT company to ship new features and adopt new technologies on the fly. In this day and age of AI, where the tech landscape shifts every Attosecond (nerd alert 🤓), keeping up can feel impossible.
But let me tell you how my years of doing goofy side-quests and building a solid understanding of the basics helped me survive my latest trial by fire.
One day, my manager came to me with a seemingly simple request: "Hey, we have data in our Power BI report and we need to extract it."
Me, who was hearing the name "Power BI" for the second time in my entire life, immediately said, "YES, ABSOLUTELY!" with a giant grin on my face 😄 and crippling anxiety in my head 😩.
I had absolutely no idea how to do this. After a frantic research session, I came up with a very dumb combination of Power Automate and DAX queries. Why was it dumb? Let me explain.
Initially, the dummy data wasn't massive. Armed with my two days of "super-knowledge" in DAX, I used the Performance Analyzer to extract the DAX queries from the report and tweaked them. I had multiple tables to pull from, and absolutely nobody had access to (or knowledge of) the actual source of truth for the data.
So, I went rogue. I extracted all the DAX queries, created a .txt file for each one, and stored them as blobs in an Azure Storage Account container. Then, using Power Automate, I fetched those DAX queries, ran them to extract the data, generated CSV tables, and dumped those files into a different Azure container.
Problem solved, right? Wrong.
A month later, they loaded the actual production data. Almost immediately, I got smacked in my handsome face by Microsoft.
It turns out the DAX query API has a hard execution limit of fetching 100,000 rows or 15 MB at a single time. My 200K+ rows were violently getting truncated down to 16K.
But I’m stubborn when it comes to technology. My brain started revving at over 9000 RPM (I get it, Vegeta). If the API wouldn't give me the data all at once, I was going to force Power Automate—an orchestrator tool—to moonlight as an ETL tool.
I did what any sane (read: desperate) programmer would do: I started paginating the data. There was no Index column in the tables, so I had to inject one using the RankX function in DAX. I filtered the Index from 0 to 5,000, pulling the data in bite-sized chunks. I appended these chunks into a giant string variable in Power Automate (the first chunk with headers, the subsequent ones without).
With a smug 😏 look on my face, I had cleared the hurdle.
Then, the data grew. It skyrocketed to over 1 million rows—amounting to over 30 million individual data points.
This time, Power Automate flat-out ditched me. It threw an error explicitly stating I couldn't store more than 100 MB of data in a single variable. Right around the 300K row mark, my pipeline completely shattered. Meanwhile, the seniors who handed me this task (titles omitted for my own safety) started dancing on my head like Jingalala HUHU.
I realized I couldn't keep hacking an orchestrator to do a data warehouse's job. I needed to migrate to a tool actually built for handling large datasets. I decided to switch the whole architecture over to Microsoft Fabric.
This meant I had to learn an entirely new tool from scratch, under immense pressure. But here is where the payoff happened. Because of my history of constantly juggling multiple tools, frameworks, and technologies, I adapted rapidly. I rebuilt the solution and successfully delivered the requirement within the stipulated timeframe.
This exact habit of constant tinkering is what makes me intuitive enough to pick up any new technology, framework, or programming language at breakneck speed. I am a Jack of All Trades, but a Master of the Basics.
The moral of the story, and what I see every single day as a software engineer, is this: You have to be adaptable, and you have to be fast. At the end of the day, no one cares how elegant your first draft was or how many errors you hit along the way. All that counts is your ability to deliver the business requirement. That is what true High Agency looks like.
Please Sign in to add comment
Comments