Have you ever asked yourself, "how do I read in 10,000 Excel files and process them using Spark?" I hope not...it sounds like a terrible task...but in case you have, it just so happens I might have an approach your interested in.
PySpark does not …
In my most recent role, we're using Python and Spark to perform a complex ETL process and to produce data that will ultimately be used to produce some model. During this process, we were using PySpark's pyspark.sql.types.DateType to store date information. At some point, we decided that …
Like most apps on the internet, the stuff that I write at Showroom Logic has scheduled tasks that must happen in a predictable fashion. In our case, we have some reports to run and deliver. Our app, like lots of internet apps, is distributed and runs in a Docker container …
Bottom line up front:
At my place of work, I'm in charge of the Data Pipeline team. Part of this team's job is to make data retrieval and processing reliable and flexible. After doing some thinking, we realized that solutions such as …